Refining a framework for measuring qualification effects

• Project leader(s): Steven SelfMark Slaymaker
• Theme: Other
• Faculty: STEM
• Status: Current
• Dates: May 2017 to May 2022

This project follows on from our previous project¹ which focused on a new pedagogical approach (Hall and Rapanotti, 2015) implemented in three post-graduate computing modules for a recently introduced Computing qualification (F66 MSc in Computing), where the students’ own professional context of practice, rather than fictitious case studies, is used to assess their understanding of and ability to apply what is taught in those modules, as well as to develop a wide range of research and employability skills as they progress through the qualification.

The aim of the previous project was a preliminary evaluation of the pedagogical approach and the definition of a framework which could be used to evaluate its effectiveness within the qualification, with particular attention to cumulative effects along different pathways students may take, and culminating in the capstone research project module, where skills acquired through this type of pedagogical approach were assumed to be particularly relevant.

The previous project was successful in achieving the following outcomes:

1. identify a selection of relevant quantitative and qualitative data, including specific student journeys as representative case studies
2. analyse the available quantitative data for cumulative qualification effects
3. quantify the workload required by module team and tutors
4. define proof-of-concept semi-automated techniques for the extraction of relevant qualitative data from unstructured data sets, to facilitate follow-up manual analysis
5. apply the techniques to specific data sets extracted from module student fora
6. trigger pedagogical adjustments within the modules under study aimed at improving student performance and retention
7. disseminate the research via a school seminar

Moreover, we are in the process of writing the final project report and submitting an abstract to the 6th eSTEeM Annual Conference.

The combination of techniques we have developed and applied to identify, extract and analyse data for this project can be seen as contributing to an overall evaluation framework, which we aim to refine and put to test in this follow-up project, whose specific objectives include:

1. incorporate new quantitative data from more recent module presentations, and data from other modules used as benchmarks
2. expand the range of qualitative data sets with student assignments, including aggregating data for specific student journeys through the qualification, to be used as representative case studies
3. customize the classification and selection scheme used by the semi-automated techniques to the range of research and employability skills under consideration
4. apply the customized framework to the new data sets
5. establish the wider applicability of the framework

Outcomes will provide an evidence-based framework with direct benefits to:

• the teams of the modules under study, for further development and fine-tuning of the pedagogical approach;
• other module teams wanting to implement or capitalise on the pedagogical approach; or to perform similar analysis on different pedagogy;
• qualification leads, as part of wider qualifications assessments.

The aim of both the previous and this project is to develop tools and processes that enable us to evaluate this new pedagogical approach (and other pedagogical approaches) by looking at the impact on student success – student performance, retention, progression and acquisition of transferable skills.

¹ “Measuring qualification effects of a new pedagogy which embeds learning and assessment activities within each student’s rich professional context of practice”.

Hall, J.G. and Rapanotti, L. (2015), Masters-level software engineering education and the enriched student context, in 'Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on', Vol. 2, IEEE, pp. 311–314.