
==== Front
Cureus
Cureus
2168-8184
Cureus
2168-8184
Cureus Palo Alto (CA)

10.7759/cureus.66278
Medical Education
Utilisation of Learning Analytics to Identify Students at Risk of Poor Academic Performance in Medical Schools
Muacevic Alexander
Adler John R
Wong Thai Ling 1
Hope David 1
Jaap Alan 1
1 Medical Education, The University of Edinburgh, Edinburgh, GBR
Thai Ling Wong thailingwong@gmail.com
6 8 2024
8 2024
16 8 e662786 8 2024
Copyright © 2024, Wong et al.
2024
Wong et al.
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License CC-BY 4.0., which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
This article is available from https://www.cureus.com/articles/280618-utilisation-of-learning-analytics-to-identify-students-at-risk-of-poor-academic-performance-in-medical-schools
Introduction

Identifying students at risk of failure before they experience difficulties may considerably improve their outcomes. However, identification techniques can be costly, time-intensive, and of unknown efficacy. Medical educators need accessible and cost-effective ways of identifying at-risk students. The aim of this study was to investigate the relationship between student engagement in an online classroom and academic performance given the transition of many courses from in-person to online learning. 

Methods

A retrospective study was conducted on a group of 235 students from the University of Edinburgh Bachelor of Medicine and Surgery (MBChB) in Year One for eight weeks from the start of term, September 2020. Purposive sampling was used. Data were collected on total test submissions, total discussion board submissions, engagement scores, and overall exam scores. Learning analytics on discussion board engagement were collected for new medical students before they had sat any summative assessment. Tests completed, discussion board posts made, and their total engagement score were correlated with their first summative assessment scores at the end of semester one.

Results

We found a statistically significant correlation between total test submissions, total discussion board submissions, engagement scores, and overall exam scores, with small-medium effects (r = 0.281, p<0.001) (r = 0.241, p<0.001), and (r = 0.202, p<0.001). Students with more test submissions, total discussion board submissions, and total engagement had a higher overall exam score. There was a statistically significant moderate correlation between total submissions and overall exam scores (r = 0.324, p<0.001).

Conclusions

Students who had a higher number of submissions were more likely to perform better on assessments. Early engagement correlates with performance. Learning analytics can help identify student underperformance before they undertake any assessment, and this can be done very inexpensively and with minimal staff resources if properly planned.

academic performance/grades
preclinical education
online medical education
learner engagement
learning analytics
==== Body
pmcIntroduction

Online learning has become increasingly popular amongst universities, especially after the COVID-19 pandemic. The National Centre for Education Statistics (NCES) reported that approximately 10 million students attended higher-education courses online, which was projected to increase by 20.5% between 2022 and 2030 [1]. Online learning platforms allow a more detailed analysis of individual learning patterns as compared to traditional in-person teaching.

Learning analytics provide important data on learners’ engagement online and could potentially play a crucial role in improving academic performance. Newmann defined engagement as “the student’s psychological investment in an effort directed towards learning, understanding, or mastering the knowledge, skills, or crafts that academic work is intended to promote” [2]. While there is extensive research on whether student engagement improves academic achievement, the findings are mixed, with a large number of studies showing a positive correlation while others found minimal or no correlation between engagement level and academic grades [3-7]. A better understanding of the effects of engagement through online learning management can help educators design effective online learning activities and materials and introduce new policies to improve student engagement and their overall performance.

In 2019, tutorials for medical students in Edinburgh were delivered virtually as a consequence of the COVID-19 pandemic. In addition, university admissions for the term 2020/2021 based on A-Level predicted grades [8] led to some uncertainty about the students’ actual academic ability. These events, coupled with the implementation of learning analytics for online classrooms, highlighted the importance of effective use of learning analytics to potentially improve student retention and academic achievements. Furthermore, preclinical grades are predictors of academic performance in clinical years [9]. This is why it is imperative to be able to utilise learning analytics effectively by better understanding the relationship between student engagement and learning outcomes in order to help identify students at risk of poor outcomes to trigger immediate intervention in preclinical years.

To our knowledge, this is the first study in Scotland to utilise learning analytics to examine the relationship between online engagement and academic performance amongst preclinical students. The aim of this study was to investigate the relationship between student engagement in an online classroom and academic performance with the utilisation of learning analytics, given the transition of many courses from in-person to online learning. 

Materials and methods

Data collection

A retrospective study was conducted on 235 students from the University of Edinburgh Bachelor of Medicine and Surgery (MBChB) in Year One. Purposive sampling was used. Data were collected over eight weeks at the start of the course, from September 2020.

The study was conducted on all Year One students for the academic year 2020/2021 to allow early identification of students at risk of poor academic performance for early intervention. This was due to the uncertainty of students’ actual academic performance due to the change of admissions criteria, which was based on predicted A-level grades. 

The topics covered in Semester One of Year One MBChB included physiology, anatomy, clinical pharmacology, knowledge of clinical practice (KCP), social and ethical aspects of medicine (SEAM), and research and evidence-based medicine (REBM).

All tutorials, quizzes, and forum discussions were delivered online on Blackboard Learn and Blackboard Collaborate Ultra (Anthology Inc., Boca Raton, FL). Blackboard Analytics for Learn (Anthology Inc.) was used to collect data for the study [10]. Data collected included the total test submissions, total discussion board submissions, total submissions, engagement scores, and total exam scores.

Total test submissions 

During each week of the Year One MBChB Programme, weekly formative quizzes were made available to encourage active learning and revision. Unlimited reviews of the formative assessments were made available for revision throughout Semester One. There was no time limit set for the formative assessments. The total test submissions were calculated by the total number of submissions of formative quizzes attempted by each student.

Total discussion board submissions

Both students and instructors were provided with access to the Blackboard Learn Discussion Boards. Discussion boards were regularly monitored and facilitated by the respective course leads. Students were able to create posts to engage in in-depth discussions with peers and instructors regarding relevant topics. The total discussion board submissions were measured by the total number of posts created on Blackboard Learn by the students. 

Total submissions

We combined the total number of submissions made for formative tests and discussion board activities to measure the total number of submissions for each student. 

Engagement score

The engagement level of students was monitored throughout Semester One. The engagement of each student was monitored during each tutorial, based on the duration of time spent on each course and online course materials, attendance, and frequency of access to courses.

Total exam score

A summative assessment was conducted at the end of Semester One. All assessments were delivered remotely. The summative assessment tested the knowledge of the content covered during Semester One. The scores were calculated out of 90 marks, and feedback was made available to students after the assessment. 

Eligibility 

Participants of the study were newly enrolled Year One students of the MBChB Programme at the University of Edinburgh during the term 2020/2021.

Ethics

Ethical approval was obtained from the University of Edinburgh Medical Education Ethics Committee (reference number: 2020/33). Students involved in the study consented to the routine evaluation of their data. All data included in this study were anonymised before data analysis was performed. Researchers involved were not able to identify individual medical students at any point of the study. 

Data analysis 

The data collected were analysed using IBM SPSS Statistics for Windows, version 27.0 (IBM Corp., Armonk, NY). Descriptive statistics used were mean and standard deviation. A Pearson’s correlation test was used to find out the correlation between the variables and students’ performance. A scatter plot was drawn to determine the relationship between the variables and the total overall score. 

Results

Overall, a total of 235 students in Year One of the University of Edinburgh MBChB Programme were included in this study. The distribution of the overall exam score calculated using the summative assessment conducted at the end of semester one is shown in the histogram in Figure 1. The histogram shows a left-skewed distribution of the overall exam score out of 90 marks. Most students obtained a score between 70 and 80 marks. The highest score obtained was 88 out of 90 marks, while the lowest score was 49.5 out of 90 marks. 

Figure 1 Histogram of the overall exam score obtained from the semester one summative assessment

The characteristics of the total number of test and discussion board submissions, engagement score, and overall exam score are shown in Table 1. A total of 235 submissions were completed for formative tests and discussion board activities. The mean submissions for formative quizzes were 4.89 ± 2.00 submissions per student. Meanwhile, the mean submissions for the Blackboard Discussion Board activities were only 1.85 ± 1.08 submissions per student. The mean submission for the overall submissions (sum of test and discussion board submissions) was 6.73 ± 2.55 submissions per student. The mean engagement score was 105.32 ± 1.55. A total of 233 students taking the Year One exam subsequently completed the final compulsory summative assessment. The mean overall exam score was 74.40 ± 8.15 marks (out of 90 marks).

Table 1 Characteristics of total submissions, total test submissions, total board discussions, engagement scores, and overall exam scores of all included students

Variables	Total submissions	Engagement score	Overall exam score (out of 90)	
Total test submissions	Total discussion board submissions	
Total, n	235	235	236	233	
Mean ± SD	4.89 ± 2.00	1.85 ± 1.08	105.32 ± 1.55  	74.40 ± 8.15  	
Total Mean ± SD	6.73 ± 2.55	

The relationship between the total number of submissions, test submissions, discussion board submissions, and engagement with the overall exam score is demonstrated in Figure 2. The results of the Pearson correlation show a significant positive correlation between total test submissions, total discussion board submissions, and overall exam score (r = 0.281, p <0.001), (r = 0.241, p <0.001), with small effects. Students who had completed a higher number of tests or total discussion board submissions were likely to achieve a better overall exam score.

Figure 2 Scatter plot showing the relationship between the total submissions (a), total test submissions (b), total discussion board submissions (c), and engagement score (D) with the overall exam score

Similarly, there was a significant positive correlation between the engagement score and overall exam score (r = 0.202, p <0.001), with small effects. Those with better engagement scores obtained by active participation in classes, having better engagement with online course content, and spending more time on online course materials were likely to perform well in their summative assessments. 

There was a moderate correlation between total number of submissions and overall exam score (r = 0.324, p <0.001). Students who completed more formative quizzes and were more involved in the Blackboard Discussion Board activities were more likely to achieve a better academic performance in their summative assessments as compared to other students.

Discussion

The significant, weak positive correlation between student engagement and overall exam score could be explained by several factors. First, increased teacher-student interaction and interaction between peers could have improved student performance through the exchange of feedback, reduced social isolation, and an increased level of affirmation. The online learning environment encourages the exchange of feedback between learners and tutors and amongst learners themselves through collaborative discussion forums and formative test submissions. Zimbardi et al. demonstrated that time spent accessing tutors’ feedback was significantly associated with improvement of academic outcomes [11]. Similarly, peer feedback was also shown to effectively improve academic achievement by promoting interaction, engaging students in their own process of learning, and encouraging learners to better understand an examiner's perspective [12]. In addition, affirmation can be delivered through positive feedback shown to effectively improve academic performance [13]. Besides that, increased interaction in online classrooms could have contributed to overcoming social isolation. Studies have found that social isolation was more likely to negatively affect educational outcomes [14, 15]. Our study was conducted during a period when medical students in Edinburgh were legally required to self-isolate to mitigate the spread of COVID-19. In-person on-campus sports and social activities were suspended at the time. The increased interaction among peers and with tutors could have helped reduce the feeling of isolation, which in turn, positively affected educational outcomes.

Second, engagement with learning activities could be associated with the affective elements of the learning process, such as personality, motivation, effort, and self-confidence. Learners demonstrating more motivation and dedication tend to do well academically and are more likely to spend more time engaging with learning activities and therefore perform better [16]. Ditta et al.’s study with the aim to address low motivation levels found that exposure to new topics motivated students to broaden their knowledge [17]. Hence, although better academic performance could be a result of higher motivation levels, effective techniques and teaching models can also be introduced by educators to, in turn, increase the learners’ motivation. In addition, the utilisation of online discussion boards could help learners overcome social anxiety and low levels of self-confidence. For example, anonymous discussion boards within the learning management systems would encourage learners to ask questions without the fear of being identifiable. An online survey showed that students were significantly more likely to post when anonymous posting was enabled [18]. Therefore, affective elements of learning such as motivation, personality, and self-confidence could affect the learning outcome and should be considered when designing online teaching activities to improve engagement levels and academic performance. 

Third, the incorporation of practice and repetition could positively influence academic performance. Students more engaged with online learning are more likely to have repeated encounters with learning materials to facilitate their participation in online discussions, completion of formative quizzes, and review of feedback obtained. Extensive research has demonstrated spaced practice and repetition as an effective way of improving learning and educational outcomes [19]. The availability of online resources enables spaced-out repetition and familiarisation of learning materials. Furthermore, knowledge retention was associated with a deep level of understanding of a topic [20]. Recorded tutorials enable learners to revisit lectures, repeat quizzes, or exercises and revise at their own pace. This allows better comprehension of a topic and improves a learner's depth of learning.

Challenges of improving engagement in online learning environments

The challenges in implementing change to improve learners’ engagement in an online environment are manifold. Socially deprived areas with a lack of access to stable internet could impede online participation and student engagement. Several institutions have attempted to address this issue by providing financial support and scholarships to students facing barriers to accessing online education. Whilst these efforts are commendable, sustainable solutions are required to ensure equal access to education. Furthermore, the online learning environment has a negative impact on students with learning disabilities, further widening the learning gap that was already present before the introduction of online learning [21]. Personalised learning environments should be introduced to account for students of different backgrounds and learning needs. Moreover, online learning can have a long-term impact on the psychological well-being of learners [22]. The benefits of a balanced mix of online and in-person teaching should be further explored to ensure effective learning.

Limitations

We acknowledge that this study has some limitations. The rise in online educational projects, such as online tutorials on YouTube and Osmosis, also meant that we were unable to identify students who were disengaged with course activities but actively revising other online materials readily available to medical students. Besides that, this study was based on a correlation design, which does not infer a cause-and-effect relationship between engagement and academic performance. We also assumed that students present at tutorials were actively participating in sessions. Furthermore, our analytics did not record active engagement during live sessions, such as the number of questions asked or polls answered. In addition, with the shift to hybrid teaching and learning in higher education, further studies would be required to better identify the effects of engagement and academic performances in hybrid classrooms. 

Conclusions

In conclusion, the level of engagement and submissions made by students directly correlates with the overall performance of students in exams. The use of learning analytics in identifying students at risk of failure provides an opportunity for educators to intervene before students experience difficulties. While online learning has its own benefits, it is important to take into consideration the advantages of face-to-face learning and incorporate hybrid teaching where appropriate. Strategies such as incorporation of active learning, providing feedback to students, and activities encouraging active participation should be introduced for students identified to be at risk of poor academic performance. 

Disclosures

Author Contributions

Human subjects: Consent was obtained or waived by all participants in this study. University of Edinburgh Medical Education Ethics Committee issued approval (2020/33). Students involved in the study have consented to the routine evaluation of their data.

Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue.

Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following:

Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work.

Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work.

Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.

Concept and design:  Thai Ling Wong, David Hope, Alan Jaap

Acquisition, analysis, or interpretation of data:  Thai Ling Wong, David Hope

Drafting of the manuscript:  Thai Ling Wong

Critical review of the manuscript for important intellectual content:  Thai Ling Wong, David Hope, Alan Jaap

Supervision:  Alan Jaap
==== Refs
References

1 2024 online learning statistics Hamilton Hamilton I I 2024 https://www.forbes.com/advisor/education/online-colleges/online-learning-stats/
2 Student engagement and achievement in American secondary schools Newmann Newmann Fred M. Ed. Ed. 1992 https://eric.ed.gov/?id=ED371047
3 Student engagement with school: critical conceptual and methodological issues of the construct Psychol Sch Appleton JJ Christenson SL Furlong MJ 369 386 45 2008 https://onlinelibrary.wiley.com/doi/10.1002/pits.20303
4 Class attendance in college: a meta-analytic review of the relationship of class attendance with grades and student characteristics Rev Educ Res Credé M Roch SG Kieszczynka UG 272 295 80 2010 https://eric.ed.gov/?id=EJ889127
5 The effect of attendance on academic performance Mediterr J Soc Sci Guleker R Keci J 961 5 2014 https://www.richtmann.org/journal/index.php/mjss/article/view/4614
6 Minimal effects of reduced teaching hours on undergraduate medical student learning outcomes and course evaluations Med Teach Choi-Lundberg DL Al-Aubaidy HA Burgess JR 58 65 42 2020 31437065
7 Does lecture attendance matter? Some observations from a first-year economics course at the University of Cape Town S Afr J Econ van Walbeek C 861 883 72 2004
8 University admissions 2021 Hubble Hubble S. S. J. Lewis P. Bolton 2021 https://commonslibrary.parliament.uk/research-briefings/cbp-9294/
9 Predicting success in medical school: a longitudinal study of common Australian student selection tools BMC Med Educ Sladek RM Bond MJ Frost LK Prior KN 187 16 2016 https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-016-0692-3 27450570
10 Analytics for Learn 2024 https://help.blackboard.com/Learn/Instructor/Ultra/Performance/Analytics_for_Learn
11 Are they using my feedback? The extent of students’ feedback use has a large impact on subsequent academic performance Assess Eval High Educ Zimbardi K Colthorpe K Dekker A 42 2017 https://eric.ed.gov/?id=EJ1129824
12 Peer feedback improves students’ academic self-concept in higher education Res High Educ Simonsmeier B Henrike P Flaig M Schneider m 706 724 61 2020
13 Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback increases secondary students’ text revision, motivation, and positive emotions Comput Artif Intel Meyer J Thorben J Schiller R 100199 6 2024 https://www.sciencedirect.com/science/article/pii/S2666920X23000784
14 Social isolation and achievement of students with learning disabilities Soc Sci Res Bruefach T Reynolds JR 102667 104 2022 35400383
15 Comparing social isolation effects on students attrition in online versus face-to-face courses in computer literacy Issues in Informing Science and Information Technology Ali A Smith D 11 20 12 2015 https://iisit.org/Vol12/IISITv12p011-020Ali1784.pdf
16 The importance of students’ motivation for their academic achievement - replicating and extending previous findings Front Psychol Steinmayr R Weidinger AF Schwinger M Spinath B 1730 10 2019 31417459
17 Exposure to information increases motivation to learn more Learn Motiv Ditta AS Strickland-Hughes CM Cheung C Wu R 101668 72 2020 32934422
18 "I'd be so much more comfortable posting anonymously": identified versus anonymous participation in student discussion boards Australas J Educ Tech Roberts LD Rajah-Kanagasabai CJ 452 29 2013 https://ajet.org.au/index.php/AJET/article/view/452
19 Spaced repetition promotes efficient and effective learning: policy implications for instruction Policy Insights Behav Brain Sci Kang SHK 12 19 3 2016 https://journals.sagepub.com/doi/abs/10.1177/2372732215624708
20 Improving learning outcomes: unlimited vs. limited attempts and time for supplemental interactive online learning activities J Curr Teach MacKenzie LM 36 45 8 2019
21 Online learning for students with learning disabilities and their typical peers: the association between basic psychological needs and outcomes Learning Disabilities Research and Practice Goegan LD Daniels LM 140 150 37 2022 https://onlinelibrary.wiley.com/doi/full/10.1111/ldrp.12277
22 The impact of long-term online learning on social anxiety and problematic smartphone use symptoms among secondary school students with different levels of fear of missing out: eEvidence from a symptom network and longitudinal panel network analysis J Behav Addict Tao Y Tang Q Wang S 102 119 13 2024 38206330
