Evaluating and Evidencing Impact: Academic Enhancement Toolkit

Introduction

The purpose of this chapter is to provide some practical guidance on the evaluation of Teaching, Learning and Assessment (TL&A) enhancement and innovation projects, initiatives and inquiries. Such evaluation in turn support scholarly reflection, planning, enhancement and dissemination through academic and other outputs. It is one of a series of chapters that make up the Academic Enhancement ToolKit that will be published throughout 2024/25.

This guidance appreciates that providing a consistently high quality learning experience is a collective endeavor underpinned by shared values and a common purpose and is dependent on individual and collaborative activity. It also assumes that evaluation (strategy) planning will form part of / be included in the project planning process for any intervention / project. Thus this guidance will complement / be used alongside the usual project planning processes.

Principles

The following tenets underpin this guidance, evaluation activity should:

Evidence the impact on student learning and involve students
Sustain engagement with scholarship in all forms
Align with institutional processes, strategies and priorities
Apply the fundamentals of research design
Facilitate collaboration, dissemination and mutual support.

In order to implement these principles a framework of evaluation practices is provided that includes how to address the following areas:

What are you trying to achieve (Theory of Change)? E.g. enhance student satisfaction and engagement with feedback
Why you are trying to achieve it? E.g. students’ (level of) engagement with, and use of, feedback impacts on student learning and outcomes and we have evidence e.g. via responses to NSS assessment and feedback questions that a significant number of students don’t find feedback helpful in improving their work. We want to address this situation.
How are you going to do it? E.g. project to increase student ‘feedback literacy’ to try and promote and support effective utilisation of feedback
How do you know you have done it immediate, medium and longer term e.g. BSS or NSS results; use of research instruments; student data
Taking a scholarly / evidence based approach: has someone solved this problem before? Can you apply their solution to your context? Will you do something new? (applying / integrating existing scholarship or discovering new knowledge).
When to go through Research Ethics approval processes.

The principles and linking evaluation practices outlined below form the heart of this evaluation guidance. It is important to note that all the principles, and in particular the associated evaluation practices, are not intended to be read and applied in a linear or sequential fashion, but to be taken together and applied holistically to your evaluation strategy and activities. That is, all should be read, considered and applied before determining and finalising your evaluation strategy.

Principles and corresponding evaluation practices

Principle	Evaluation practice
1. Evidence the impact on student learning and involve students.	Evaluation practice 1.1: Purposes of evaluation including the role of students to achieve that purpose. Evaluation practice 1.2: Planning for evaluation (establishing a ‘theory of change’).
2. Sustain engagement with scholarship in all forms.	Evaluation practice 2.3: What is the ‘level of rigour of inquiry’. Evaluation practice 2.4: Evidence-informed approach to project inception.
3. Align with institutional processes, strategies and priorities.	Evaluation practice 3.5: Anticipate potential consequences including ethical and organisational issues.
4. Apply the fundamentals of research design.	Evaluation practice 4.6: Define the scope of intended evaluation including inclusion/exclusion criteria. Evaluation practice 4.7: Describe the evaluation criteria, questions, and methods. Evaluation practice 4.8: Assign evaluation roles and responsibilities and associated resource needs.
5. Facilitate collaboration, dissemination and mutual support	Evaluation practice 5.9: Sharing and communicating findings with all stakeholders.