Measuring qualification effects of a new pedagogy which embeds learning and assessment activities within each student’s rich professional context of practice

Project leader(s): Lucia Rapanotti
Theme: Other
Faculty: STEM
Status: Archived
Dates: January 2016 to May 2017

We report on the outcomes of an eSTEeM project focused on a pedagogical approach implemented in three post-graduate modules for a recently introduced Computing qualification (F66 MSc in Computing), where the students’ own professional context of practice, rather than fictitious case studies, is used to assess their understanding of and ability to apply what is taught in those modules, as well as to develop a wide range of research and employability skills as they progress through the qualification. At its core, the approach blends the learner’s study and their rich real-world context of practice in a form of situated learning backed up by authentic assessment.

The project, which run over 12 months, conducted a preliminary evaluation of the approach and contributed to the definition of a generic framework to be used to evaluate its effectiveness within the qualification, with particular attention to cumulative effects along different pathways students may take, and culminating in a capstone research project module, where skills acquired through this type of pedagogical approach are particularly relevant.

A first main project objective was to compare and contrast contextualisations of the pedagogy across modules. This objective was fully achieved and identifed subtle differences in interpretation of the pedagogy by the different modules and related implementations, and how subsequent adjustments were made as a result of feedback from year-on-year presentations. The project was able to surface and share good practices, which have already influenced further adjustments of teaching and assessment across the modules. The latter has also contributed to meeting another project objective – that of identifying opportunities for sharing resources and practices.

Another project objective concerned quantifying the impact of the pedagogy on module teams’ and tutors’ workload and practices. In terms of central staff workload and overall financial contribution the modules under study appear to perform better than comparators that adopt a traditional pedagogy, and are in line with another comparator that adopts a situated learning pedagogy. However, at least qualitatively, associate lecturers have reported that marking assignments is perceived as more demanding, although also more rewarding.

The more challenging project objective was to quantify effects through pathways on students’ performance, satisfaction and retention, particularly for students who have completed multiple modules. Such an objectives was broken down into further sub-objectives. Firstly, we identified a selection of relevant quantitative and qualitative data, including specific ‘student journeys’ as representative case studies. Data included both standard key performance indicators around registration, retention, progression, students satisfaction, etc., and qualitative data sets from module fora and students’ assignments. Next, we performed an in-depth analysis of the quantitative data. Briefly, the modules under study perform at least as well as the comparators in terms of students’ attainment and satisfaction, but retention appears to be lower. Possible reasons have been considered, which have already triggered further investigation and pedagogical adjustments aimed at improving students’ retention: the outcome will be monitored in future module presentations.

In parallel to the quantitative data analysis, we turned our attention to the qualitative data sets. It became soon apparent, however, that the amount of available data would have been prohibitively time consuming to analyse manually, and even manual sampling would have been unachievable with the project timescale. Therefore, instead of attempting a partial qualitative analysis, we looked for more creative ways to tackle the problem and developed semi-automated tools for data selection, extraction and classification, based on Natural Language Processing techniques, with the potential to increase the efficiency of future manual analysis quite significantly. A proof-of-concept framework was developed and tested on a data set consisting of module forum posts. The intention was then to follow this activity up with a manual analysis of the extracted data, something we were unable to accomplish within the timeframe of the project. Although this was a departure from the initially envisaged project approach, we feel it has led to a positive and promising outcome and paved the way for future research.

It is worth noticing that access to the data sets of interest was very difficult. While within the OU we are awash with management information and data analytics based on individual modules, there appear to be no automatic tools to extract data for comparisons between modules. Moreover, data are not consolidated in an easy-to-access data warehouse. Instead, for this project data had to be extracted manually from a variety of systems, often requiring a lengthy process to identify stakeholders with the authority to access such systems. We feel this should be a lesson for the institution: better processes and tool support for data extraction should be put in place to facilitate this type of scholarship. It is certainly the case that precious time was wasted on this project due to the overhead of locating and accessing the raw data.

Overall, we think that the combination of techniques we have developed and applied to identify, extract and analyse data for this project can be seen as contributing to an overall evaluation framework, which could be used not just for this project, but in general for studies of a similar nature. That said, the framework still needs adjusting and evaluating, so that further work is required. In particular, this research will benefit from a follow-up project, whose specific objectives may include:

incorporate new quantitative data from more recent module presentations, and data from other modules used as benchmarks
expand the range of qualitative data sets to include open ended answers in survey data and student assignments, including aggregating data for specific student journeys through the qualification, to be used as representative case studies
customize the classification and selection scheme used by the semi-automated techniques to the range of research and employability skills under consideration
apply the customized framework to the new data sets
measure the accuracy of precision and recall
establish the wider applicability of the framework

Outcomes from this new project would provide an evidence-based framework with direct benefits to the teams of the modules under study, for further development and fine-tuning of the pedagogical approach, as well as to other module teams wanting to implement or capitalise on the pedagogical approach, or perform similar analysis on different pedagogy.