How Early can we Identify At-Risk Students? An Analysis of LMS Interactions

Abstract

The switch to online learning in higher education brought about by the Covid-19 pandemic has had lingering effects – most notably, continued higher levels of usage of learning management systems (LMS) such as Moodle for assessment and sharing of course materials. This has enhanced the potential for learning analytics even for courses that are delivered in a face-to-face mode. This is because the design of the course page on the LMS and how it is utilized for assessments over the semester necessarily affect the nature of student interactions with the LMS. There is already a sizeable literature that links student interactions with the LMS, selected student characteristics, and learning outcomes, highlighting that it is indeed possible to detect at-risk students using data sources such as course logs and click streams. However, there is less research on how early a student who is at risk of not completing or failing the course can be detected. This paper uses LMS logs, student characteristics, and learning outcomes of six cohorts of undergraduate students (over 500 students in total) taking a compulsory second-year module in a Sri Lankan university to detect the earliest point in the semester at which at-risk students can be identified. Due to the weekly modeling structure, the dataset expands to over 8,000 records, with each entry corresponding to a unique combination of student index number and week number. This paper employed a cumulative modeling approach, where several machine learning models including Random Forest, Decision Tree, Support Vector Machine (SVM), and K-Nearest Neighbors (KNN) are assessed for performance. Random Forest consistently outperformed other models, achieving an accuracy of 78.51% in Week 16. Notably, performance metrics stabilized above 70% by Week 8, suggesting it as the optimal point for early prediction. The analysis revealed that prior academic performance and consistency of LMS engagement were stronger predictors than total LMS clicks. These findings support the development of data-driven early warning systems tailored to the Sri Lankan higher education context, emphasizing the value of consistent behavioral monitoring and historical academic data for effective intervention strategies and it provides insights on how effectively utilizing an LMS can improve learning outcomes even for courses that are offered in face-to-face mode.