Student engagement analytics: how to move from baffled browsing to concerted action

At the Open University we teach our students at a distance through a Virtual Learning Environment (VLE), with each module having its own website and much of the tuition happening online. When I started as an Associate Lecturer (AL) I explored the resources in the AL-facing part of my module’s VLE and remember stumbling over a very unexciting button named Early Alerts Indicators which I did dabble with occasionally. This turned out to be the Student Engagement Dashboard, one of the tools that helps ALs and managers understand how much our students are engaging with the VLE, through predictive analytics. In reality my dabbling with the dashboard was really an ironing avoidance activity, and it was done with no real purpose, or meaningful outcomes. Although I was impressed by the dashboard’s traffic light system, I was never really sure what to do next. 

In my more recent role as a manager I have dedicated my time to demystify things that used to baffle me when I was an AL, and the dashboard was one of those things. Along with my colleagues Tracey Elder and Kate Hammond, we set about trying to create something tangible, in the form of a toolkit that would move ALs from baffled browsing to concerted action. With this in mind we initiated the project by being trained by the dashboard’s creators. They showed us its full potential, including how many online lectures students signed up for and attended, the number of times students clicked through the module website, and the likelihood of them submitting assessments based on current levels of activity. Armed with newfound knowledge we set about creating our toolkit, bespoke for our level 1 psychology module. The toolkit serves two purposes. Firstly it offers a framework through a schedule of dates, pertinent to this module, indicating when would be ideal times for ALs to check in and see how their students are getting on. The second function of the toolkit is to offer templated emails suggesting interactions with their students based on the scheduled reason for checking the dashboard.  

Having created the toolkit we needed to get ALs involved in using it, so offered training to all the ALs responsible for delivering the module during the 2021 academic year.  The training involved using screenshots of the dashboard, case studies and live dashboard action to showcase what it could do, and how an AL might use it differently in various types of situations. During the training we showed ALs where to find the toolkit and uploaded a recording of the training to the module’s Tutor Website for those who could not attend. 

The scholarship aspect of the project was to evaluate the effectiveness of the intervention, which we did via an online survey of all ALs responsible for teaching this module, questioning them on their awareness, use of and thoughts on the dashboard, training and toolkit. The findings are based on the 35 responses received (10% response rate).  

Associate Lecturers’ interactions with the dashboard 

As can be seen, most ALs are aware of the dashboard’s existence, but very few use it regularly, suggesting the need for guidance.  

Associate lecturers’ interactions with the training and toolkit 

Half of those surveyed had attended or watched the training video, which means that potentially half of the ALs on this module were therefore unaware of the toolkit. 

Solutions  

We believe, from both qualitative and quantitative data analysed, that those who engaged to some degree with the toolkit and training, were better equipped to support their students, and as such we intend to share this message more widely with Module Teams to encourage a greater adoption of the dashboard, with a view to increasing retention, through more targeted support of students. 

For more information, please contact us on CathySchofield@open.ac.uk or Tracey.Elder@open.ac.uk,  we are always happy to share.