Evaluation practice 3.5

Your theory of change (see evaluation practice 1.2) enables you to visualise the issue in hand and identify required inputs and outputs, as well as expected outcomes and anticipated impact(s) all at the outset of your planning.

It is always advised to start with an evaluation strategy when planning any kind of initiative or intervention. Having an evaluation strategy gives visibility of the ‘pre’ intervention (or baseline data) you need to have available (or collect) in order to be able to evaluate the impact of your intervention i.e. what ‘pre’ and ‘post’ intervention data you need to have available to capture (any) impact. See evaluation practice 4.6 and practice 4.7 for discussion on types of data you might collect according to your intervention. Evaluation undertaken as a ‘late addition’ to the project, rather than being planned ahead will leave you unsure or unable to establish some of the impact of your intervention.   

Having identified the intended outputs, outcomes and impact it is then a matter of determining the data and evidence (see evaluation practice 4.6) that will demonstrate if the desired outputs, outcomes and impact have been achieved. Where possible this should draw on existing data sets available within the organisation (see guidance in evaluation practice 4.7). So there are choices to be made here informed by your theory of change.

At this stage, and the focus of this evaluation practice, ethical issues including General Data Protection Regulation (GDPR) compliance as embodied in the Data Protection Act 2018, and organisational issues will need to be considered and you will likely need to revisit these once the scope and evaluation criteria, questions and methods have been determined (see again evaluation practice 4.6 and practice 4.7).

When planning your evaluation strategy it is important to anticipate potential consequences, as well as check your understanding of the existing sources of data available to you.

For example this link gives access to a range of student survey data (NSS, PTES, BSS etc by programme and faculty) available to relevant staff at Oxford Brookes University. An Access and Participation Plan (APP) data dashboard is in development and will be made available by Oxford Brookes University’s SCAP team. There are also alternative research designs you can adopt which are not dependent on such data, see e.g. this guidance on evaluation with small cohorts including how to evaluate the impact of interventions on target student groups.  

The following questions are designed to assist you in anticipating potential consequences and then to plan your evaluation accordingly:

• Who (including for example student cohorts or year groups not included in the intervention(s)) might be excluded or miss out because of the intervention, evaluation and/or findings?
• How will any unintended organisational consequences be uncovered or monitored and treated? 
• What might be the wider (organisational / sector / professional community) response to the findings?

Again the following questions are designed to assist you in anticipating potential consequences/issues and then plan your evaluation accordingly:

• How will data be stored in accordance with the Data Protection Act 2018?
• How are your ensuring?
  • Informed consent from those involved in the evaluation
  • Harm minimisation and appropriate confidentiality

• Have you considered if there are any possible intellectual property issues?

If you plan to disseminate findings externally you need to follow your faculty research ethics procedure. Please check this research ethics guidance first, and then consult your school or faculty research officer if in doubt. Please note you will need to consider whether you are planning to publish before the project/intervention starts and build in plenty of time for ethics to be approved before commencing your intervention.