Evaluation practice 2.3

What is the ‘level of rigour of inquiry’

Office for Students (OfS) typology establishes the following broad-based standards of evidence and evaluation:

	Description	Evidence Used	Possible Claims
Type 1: Narrative	The evaluation provides a narrative and coherent theory of change to motivate the selection of activities in the context of a coherent strategic approach.	Evidence of impact elsewhere and / or in the research literature on effectiveness, or from your existing evaluation results.	Can provide a coherent explanation of what and why. Claims are research-based.
Type 2: Empirical	Evaluation collects data on outcomes and impact and reports evidence that those receiving an intervention might have better results, although this does not establish any direct causal effect.	Qualitative and / or quantitative evidence of a pre / post treatment change or a treatment / non-treatment difference.	Demonstrates that interventions might be associated with potentially beneficial results.
Type 3: Causal	Evaluation methodology provides evidence of causal effect of an intervention.	Qualitative and / or quantitative evidence of a pre / post treatment change on a treated group relative to an appropriate control or comparison group, using an appropriate and robust research design process.	Intervention causes improvement and demonstrable difference, using a control or comparison group.

Table: Office for Students standards of evidence in evaluation

In evaluation practice 4.7 there is an example of type 2: empirical evaluation for a ‘feedback literacy’ initiative. A type 3: causal would need to be justified in terms of the resource (see evaluation practice 4.6) and would normally require ethical approval and if in doubt please contact your school or faculty research officer. See also guidance in evaluation practice 3.5 to help you.

It is also useful to understand where the findings from the evaluation process undertaken are located in terms of the ‘level of rigour’ of inquiry:

Level of Inquiry	Attributes
Level 1: [Good] or [Outstanding] teaching.	Involves the use of good (or outstanding) content & teaching methods.
Level 2: Scholarly teaching.	As above and classroom assessment and evidence gathering, informed by best practice and best knowledge, inviting of collaboration and review.
Level 3: Scholarship of Teaching	Is public and open to critique and evaluation, is in form that others can build on…..
Level 4: Rigorous Research in Education	As for level 3 but it includes a few unique components such as beginning with a research rather than an assessment question (see Borrego, 2007 p.94 for further details)

Table: Levels of rigour in inquiry (based on Borrego, 2007).

Please note within a professional learning community such as those at Oxford Brookes University who are evaluating their TL&A enhancement and innovation projects, it is not expected (or desired) that each project / individual engage with all levels at any one time rather the professional learning community as a whole engages with all levels and shares their learning to promote sustainable and sustained enhancement activity.

“A Professional Learning Community is characterised by shared values and a collective sense of purpose. Its learning is fuelled by collective enquiry, teamwork and the study of evidence. Leadership is distributed and there is collective responsibility for student achievement. Most important, though, there are supportive conditions to foster and facilitate learning – including high mutual expectations and accountabilities.” (Jackson and Tasker, 2003, [no pagination]).

Evaluation practice 2.3

What is the ‘level of rigour of inquiry’

References