
==== Front
Heliyon
Heliyon
Heliyon
2405-8440
Elsevier

S2405-8440(24)13650-X
10.1016/j.heliyon.2024.e37619
e37619
Research Article
Exploring the effects of multiple online interaction on emotions of L2 learners in synchronous online classes
Hu Xuhui xuhuihu@stu.edu.cn
ab
Gao Jian jgao@stu.edu.cn
a⁎
Romano Daniela Daniela.Romano@dmu.ac.Uk
cd1
a School of International Education, Shantou University, China
b School of Chinese as a Second Language, Peking University, China
c Institute of Artificial Intelligence, De Montfort University, United Kingdom
d Department of Information studies, University College London, United Kingdom
⁎ Corresponding author. School of International Education, Shantou University, 243 Daxue Road, Jinping District, Shantou, Guangdong Province, China. jgao@stu.edu.cn
1 Co-corresponding author with equal contribution.

07 9 2024
30 9 2024
07 9 2024
10 18 e3761929 4 2024
30 8 2024
6 9 2024
© 2024 Published by Elsevier Ltd.
2024

https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
The decline in both the quantity and quality of interaction has emerged as a notable challenge in online learning. However, the definition of interaction quality remains unclear. This study clarifies it as a decrease in the breadth of interaction, which refers to interaction that can only cover a smaller number of learners. To address this, a synchronous interaction modality, termed Multiple Online Interaction (MOI), based on Zoom's interactive tools, was introduced. In a quasi-experiment involving 58 Chinese L2 learners (with 30 beginner and 28 intermediate students), emotions were assessed using the Brief Mood Introspection Scale (BMIS) while real-time emotional dynamics were revealed through the analysis of 5129 facial expression images during a 35-min synchronous class. MOI participants reported higher levels of Lively and Happy but also experienced more Nervous and less Calm. These emotional dynamics, tracked through expression recognition technology, demonstrate that MOI's impact is primarily observed during the first Grammar & Practice section of the teaching. The empirical findings of this study provide practical insights for educators aiming to conduct effective online teaching in the future.

Keywords

Second language teaching
Synchronous online teaching
Classroom interaction
Student emotion
==== Body
pmc1 Introduction

In second language (L2) classrooms, teacher-student interaction is pivotal for learners [1]. However, the COVID-19 outbreak has led to a dramatic change in the interactive modality in L2 teaching, particularly in the synchronous online environment [2]. Researchers have increasingly recognized that online teacher-student interaction differs from that in traditional offline settings [3]. This new mode of interaction may potentially contribute to additional negative emotion such as anxiety and stress among students [4].

Emotions play a substantial role in language learning [5], and this is especially pronounced in the context of online L2 classes due to the physical separation between teachers and students. The manner in which interaction unfolds can trigger an array of emotions among students, significantly influencing their L2 learning outcomes [6]. Despite extensive investigations into teacher-student interaction and learner emotions as separate entities in the field of second language acquisition (SLA), their interconnectedness within the online teaching context has remained relatively uncharted. In addition, most research in this area has been limited to case reports or classroom observations [7,8], with few experimental or quasi-experimental studies investigating diverse interaction approaches and their effects on learners. This paucity in research approaches might impede the establishment of empirically supported methods for effective online teaching within the broader context of future online education.

Therefore, this study aims to examine the impact of specific interaction approaches on students' emotions in synchronous online L2 classes through a quasi-experiment. The results of this research endeavor are poised to offer empirical support for the significance of online interaction and, in turn, contribute to the enrichment of pedagogical practices within the sphere of distance education.

1.1 Interaction in synchronous online learning

1.1.1 Definition of synchronous online learning and interaction

Synchronous online learning is a mode of education where students and teachers, though physically separated, engage in planned activities to transmit course content. This process often involves real-time, two-way communication facilitated by multimedia software [9]. Its widespread adoption during the pandemic also presented challenges for instructors. Consequently, numerous studies have delved into elements that could contribute to online learning. Interaction and emotion are two recurring themes in this line of research, which will be reviewed in detail in the following.

Interaction refers to the verbal and nonverbal communication between teachers and students, and among learners in the classroom [10]. It is a fundamental aspect of education, particularly in language learning [11]. Of different types of interaction, teacher-student interaction in L2 classes is important, which enable educators to initiate engagement and comprehension assessment, even across geographical distances, stimulate participation and thereby enhancing educational outcomes [12].

1.1.2 The challenge of interaction in synchronous online learning

One prevailing issue identified in online L2 learning is probably insufficient interaction [13,14], and scholars have attempted to explain this issue further, considering both the quantity and quality of interaction. Harsch and colleagues found a decline in interaction in online environments compared to face-to-face settings, citing the absence of nonverbal cues, misunderstandings due to simultaneous speaking, and limited proactive student participation [15]. Hu, who observed Chinese L2 classes, noted that online classes had a similar number of teacher-student interactions as offline classes [8]. However, his study revealed that online classes exhibited more limited interaction forms initiated by teachers. This limitation was attributed to network delays on students' ends, hindering synchronous responses. Additionally, when students' cameras were off, it became difficult for teachers to discern technical issues from students' struggles, causing hesitation in seeking volunteers. Consequently, teachers often resorted to direct methods like roll call or providing answers themselves [8].

There remains a significant gap in defining and measuring interaction quality in synchronous online teaching processes. Traditional assessments of interaction quality rely on subjective feedback [15] and in-classroom observations [8], which often fail to identify the underlying causes of constrained online interaction. Therefore, there is a pressing need for further research to refine the definitions of interaction quality and explore the multifaceted factors influencing it in the context of online learning.

1.1.3 The proposal of multiple online interaction

To delve deeper into the factors influencing online interaction, we first propose a conceptualization of quality of online interaction based on two dimensions: breadth and depth, By examining these dimensions, we elucidate the underlying factors contributing to the decline in interaction quality observed in previous literature on online teaching. Building upon this conceptual foundation, we consolidate prior empirical instructional methods into the approach of Multiple Online Interaction. In this context, Breadth refers to how many learners are involved in teacher-initiated interaction within a given period, while depth relates to the level of engagement or communication concentration in that interaction.

Online interaction, distinct from face-to-face interaction, often exhibits limitations in terms of breadth. This limitation can potentially impede interaction quality, as online settings may lack the richness of nonverbal cues found in traditional classrooms [15]. Nonverbal communication, such as eye contact and body language, is considered a convenient and efficient channel for the teacher and students to communicate their thoughts and emotions, and teachers' nonverbal behaviors can promote students’ learning motivation and attention [16].

Thus, we categorize nonverbal communication, such as eye contact and body language, as a form of interaction characterized by high breadth and low depth. Conversely, when teachers call out specific students' names, it exemplifies a narrower breadth and higher depth of interaction. This perspective helps elucidate the findings of Hu's classroom observation statistics [8]. Specifically, despite teachers' efforts to generate more teacher-student interaction in online teaching to match the level of interaction in face-to-face settings, the limitations of the online environment and teachers' experience [7] restrict the types of interaction to narrower breadth interactions (e.g., roll call), consequently reducing opportunities for a wider range of students to engage with the teacher during class time.

Synchronous online teaching, however, offers advantages through the utilization of various online tools that can expand the breadth of interaction and enhance teaching effectiveness [2,17]. For example, Cheung reported a case of a veteran primary school teacher who engaged in multimodal exchanges with his English L2 students during synchronous online English lessons [7]. The teacher adeptly leveraged the affordances provided by Zoom, such as chatrooms, gesture responses, annotations, and polling, to elicit a multitude of nonverbal and expanded verbal responses from students. This phenomenon is aligned with mode-switching in synchronous virtual classroom interactions, encompassing variations in spoken and written expressions [18]. Scholars further contribute to this discourse by using designedly incomplete utterances in online synchronous course IRF sequence [19]. They facilitate collaborative writing on a shared screen to engage student participation, allowing the teacher to pose identical questions to multiple students and thereby enhancing the breadth of interaction.

Despite sporadic studies showcasing these benefits, critical experimental evidence is lacking to confirm the impact of different online interaction modes on learners. To strengthen the validation of these interaction modes through experimental methods, we propose synthesizing prior case studies of online interaction into the Multiple Online Interaction (MOI) approach, which utilizes multimedia functions such as typing and polling [2,7,18,19]. The MOI approach incorporates multimedia functions, such as typing and polling, with the aim of broadening the breadth of interaction quality within online classes. Specific examples and implementations of MOI will be presented in the Methods section.

1.2 Students’ emotion in L2 learning

1.2.1 The role of emotions in L2 learning

Emotions are transient mental states that manifest in physiological and behavioral responses [20]. In the field of language learning, a substantial body of research has established the influence of emotions on students’ learning outcomes, attitudes, and well-being [6,21,22]. Early studies have primarily focused on examining negative emotions, such as anxiety, which is a multifaceted construct encompassing self-perceptions, beliefs, emotions, and behavior [23]. Overwhelming anxiety has been found to have detrimental effects on language learning and communication [24].

In contrast, positive emotions, such as hope, interest, confidence, and enjoyment, have been considered as facilitators of learning [21]. Specifically, enjoyment, defined as a state in which psychological needs are fulfilled [25], has been found to significantly influence L2 learning outcomes, attitudes, willingness to communicate, and well-being in numerous studies [21,26]. Notably, anxiety and enjoyment in L2 learning do not exhibit a see-saw relationship. An increase in one does not necessarily lead to a decrease in the other [22,25].

Given the significant role of emotions in learning, researcher has explored teachers' behaviors that may influence students' emotions. Alrabai found that teacher interventions integrating positive and negative emotions led to improved motivation and reduced anxiety among L2 learners [27]. However, it's unclear if these effects hold in synchronous online classes, where reduced emotional resonance due to disembodiment may weaken emotions [28]. Though Başal and Eryılmaz have suggested that teacher interaction and collaboration in synchronous online teaching could potentially alleviate students' negative emotions [29], further empirical research is warranted to substantiate these claims.

In summary, previous research has focused on specific emotions like enjoyment and anxiety but has left a gap regarding how teachers' behaviors or interactive modes may impact a broader range of student emotions, particularly in synchronous online environments.

1.2.2 The assessment of emotion and facial expression recognition

Most previous methods for assessing emotions focus on developing independent questionnaires or interview that capture emotions through state-based descriptions of enjoyment or anxiety [26,30,31]. Some studies have utilized psychological research to conduct comprehensive evaluations of various learner emotions. For example, Mayer and colleagues [32,33] developed a concise 16-adjective scale (e.g., “Happy,” “Sad,” “Nervous”) for participants to rate the intensity of these adjectives. While this scale has found success in various fields, particularly online or virtual environments [34], it has not yet been applied in the context of L2 learning. It's worth noting that Mayer used the term “mood” instead of “emotion” in their work. Although mood and emotion have distinct conceptual meanings, they are frequently used interchangeably [35]. Other researchers highlight that mood is a long-lasting affective state (e.g., cheerful, melancholic), while emotion is a more nuanced, short-lived affective phenomenon (e.g., joy, sadness) [36]. As the types of mood described in Mayer's study closely align with the definitions of emotions in the L2 field, we use these terms interchangeably in this study.

However, self-report methods, commonly used to measure emotions in research, can be biased by cognitive factors and social desirability [37]. This susceptibility means that individuals may tend to provide socially desirable responses, which can compromise the reliability of the results [38]. To mitigate this, facial expression recognition has been employed to complement emotion measurement, given the established link between facial expressions and emotions [[39], [40], [41]].

Early facial expression research relied on human observation and coding [42] to enhance objectivity beyond self-report measures. Yet, subjectivity remained due to factors like observer experience and cultural differences [43]. Consequently, automated facial expression recognition technology has gained popularity due to its objectivity and consistency [44]. The Microsoft Emotion Recognition API has been adopted in various studies. It works by detecting faces in an input image, extracting facial features to overlay a geometric grid, and using classification algorithms to determine emotions based on the positions and directions of key features, providing confidence levels for each identified emotion. Kutt et al. validated its accuracy and superiority in recognizing eight basic emotions (Anger, Contempt, Disgust, Fear, Happiness, Neutral, Sadness, and Surprise) using four benchmark datasets: Cohn-Kanade, Extended Cohn-Kanade, Amsterdam Dynamic Facial Expression Set, and Radboud Faces Database [45]. Researchers have applied this technology in empirical research to uncover students' emotional states or engagement levels [43]. For example, Liu and colleagues recorded students' smiling expressions in smart classrooms and utilized them as indicators of engagement [46]. Tonguç and Ozkara utilized automatic facial expression recognition to reveal the dynamic changes in emotions of participants during an offline lecture [47].

However, these studies have primarily focused on offline classrooms. The physical distance between teachers and students, and reduced peer interaction, may influence learners' emotions in online environment [28,48]. Additionally, online learning offers certain advantages, such as eliminating the need for physical smart off-line classrooms, as cameras are already widely used in online courses. This provides a significant opportunity for the application of automated facial expression recognition technology. To the best of our knowledge, no research has employed automated facial expression recognition technology to capture students' emotions in response to specific teaching approaches in online learning environments.

1.3 This study

This study addresses gaps in existing research on synchronous online L2 learning by conducting a quasi-experimental investigation into the impact of various interaction modes on student emotions during an approximately 35-min synchronous online class, utilizing the Zoom platform. The Multiple Online Interaction (MOI) approach utilizes multimedia functionalities [2,7,18,19], such as polling and typing, within the experimental groups. In contrast, the control groups, receiving the same teaching content, employ a limited interaction modality characterized by common but narrow interactive approaches, such as roll call. The study incorporated self-reported emotions measurement through the Brief Mood Introspection Scale (BMIS) [32,33], as well as facial expressions based on the Microsoft Emotion Recognition API. The proposed research questions are as follows, with detailed methods explained in the Methods section.RQ1 Does the use of MOI in synchronous online L2 classes facilitate higher levels of self-reported positive emotions in students compared to the limited interaction modality?

RQ2 How do students' emotions vary across teaching sessions in class when comparing MOI with the limited interaction modality?

2 Methods

2.1 Participants

In this quasi-experimental study, we involved 60 learners studying Chinese as their L2, comprising undergraduate and language preparatory students from three universities in China. Participation was voluntary, and all invited participants provided informed consent. Two students were excluded due to incomplete survey responses, resulting in a final sample of 58 learners (26 males and 32 females), aged 17–25 years [M = 20.4]. Among them, 30 were classified as beginner-level students, while 28 were categorized as intermediate-level students based on their Chinese proficiency upon course enrolment. The participants had diverse first language backgrounds, including English, Arabic, Indonesian, Japanese, Lao, Urdu, Russian, Thai, Malay, Khmer, Myanmar, Pashto, French, Turkmen, Ethiopia, Bengali, Tamil, Yoruba, Odia, Persian, Hindi, and Lithuanian.

Facial expression data were gathered from 23 participants: ten from the experimental group and 13 from the control group. A smaller sample was obtained due to some students not complying with the camera request. Additionally, not all expressions were successfully recognized due to temporary absences, obstructed faces, or poor lighting. To ensure data quality, a recognition rate of at least 90 % was considered acceptable, as it is considered an efficient threshold [49]. Data from the 23 participants met this criterion, with a recognition rate of 97.29 %. Consequently, these data were used in subsequent analyses.

2.2 The design of quasi-experimental framework

To ensure the robustness of the quasi-experimental setup, a variety of strategies were employed in participant allocation, teacher assignment, and maintaining consistent curricular material (Table 1).Table 1 Participant allocation and grouping in the quasi-experiment.

Table 1Recruitment
Classes	L2 Level	Group Assignment	Number of Participants	Instructors	Mode of Interaction	
Class 1	Intermediate	Experimental	10	Teacher A	MOI	
Control	8	Teacher B	/	
Class 2	Beginner	Experimental	5	Teacher B	MOI	
Control	8	Teacher A	/	
Class 3	Intermediate	Experimental	6	Teacher B	MOI	
Control	4	Teacher A	/	
Class 4	Beginner	Experimental	10	Teacher A	MOI	
Control	7	Teacher B	/	

Participants were drawn from university classes and subjected to random allocation. Four classes were included in the study, within each class, students were randomly assigned to either the experimental or control groups using Zoom's breakout room feature.

Two teachers, Teacher A and Teacher B, with similar teaching backgrounds, employed different interaction modalities to enhance student engagement. Both instructors held master's degrees in teaching Chinese as an L2 and had approximately five years of teaching experience. To mitigate the impact of individual teaching styles on the results, two teachers employed distinct interactive modalities to instruct four separate classes in total. Each teacher instructed two classes using the MOI approach and two classes using a limited interaction model.

Notably, while the instructional content remained consistent, the primary difference was the interaction mode presented in the PowerPoint materials. In the control group, questions were verbally allocated, while the experimental group's PowerPoint included guidance notes like “Type your answer in the Chat Box”. Both teachers received training and strictly adhered to the prescribed instructional materials.

2.3 Teaching and interaction materials

In each of the four classes, encompassing both control and experimental groups, a synchronous online Chinese L2 teaching session lasting approximately 35 min (M = 35.94, SD = 3.7) was conducted. The teaching materials were consistent and followed a predetermined sequence, including Lead-in, Vocabulary, Text Ⅰ, Grammar & Practice Ⅰ, Text Ⅱ, Grammar & Practice Ⅱ, Summary & Culture. For beginner-level students, the teaching content covered “Chinese resultative and modal complement,” while intermediate-level students received an overall review of all Chinese complements.

The experimental group experienced the MOI modality, encompassing three types of questions: Volunteer & Roll Call, Typing, and Polling. In the Volunteer & Roll Call, the teacher posed a question to the entire class and waited for volunteers or called on a student if no one volunteered. The Typing required students to respond by typing and sharing the sentences they constructed based on provided pictures and grammar. Polling were also included, where students selected the correct answer from True/False or A/B/C/D options. In contrast, the control group exclusively used the Volunteer & Roll Call for instruction. A total of 18 questions were designed for the teaching section in both groups.

Fig. 1 illustrates an example of the application of MOI in the Lead-in section. The teacher presents interactive content on a PowerPoint slide, showing two images along with the question displayed in both English and Chinese: “Which of the following is a Chinese wedding?” The teacher then uses the polling feature in Zoom to send the question to each student, who selects either option A or B within the given time frame. Finally, the teacher anonymously displays the collective responses of the class and proceeds with the lesson based on the students’ answers.Fig. 1 Application of MOI in the lead-in section.

Fig. 1

2.4 Measurements

Background Questionnaire: The background questionnaire comprised items pertaining to age, gender, nationality, first language, and Chinese proficiency level.

Emotion Questionnaire: The emotion questionnaire employed in this study was the 16-item adopted from the third version of BMIS [32], which assessed various emotions such as Lively, Happy, Sad, Tired, Caring, Content, Gloomy, Jittery, Drowsy, Grouchy, Peppy, Nervous, Calm, Loving, Fed up, and Active. The English version of the BMIS was administered online, and responses were recorded on a 4-point Likert scale ranging from 1 (definitely do not feel) to 4 (definitely feel).

Facial Expression Recognition: Facial expressions during class were recorded, and images were generated every 10 s. The images were processed using an application developed for this study, which utilized Microsoft Emotion Recognition API to detect the strengths of eight basic emotions (Anger, Contempt, Disgust, Fear, Happiness, Neutral, Sadness, and Surprise) automatically, with scores ranging from 0 to 1. This methodology was inspired by previous studies that used facial expression recognition to capture emotional changes in offline lectures [47].

2.5 Procedure

Pre-Study Preparation: A week before the official study commenced, participants who provided informed consent received an e-link to complete a background questionnaire. The questionnaire and instructions were provided in English, which was also the medium language for all participants' Chinese L2 classes.

The quasi-experiment comprised the following steps and was repeated four times, twice for beginner classes and twice for intermediate classes:

Introduction and Emotional Baseline: All students watched a 7-min 30-s documentary clip titled "China's Mega Projects: Manufacturing.” to minimize potential emotional interference that students might have brought with them before the class and to establish a consistent emotional baseline for all participants [50].

Pre-Instruction Emotion Assessment: After watching the documentary clip, participants accessed the BMIS through an e-link to assess their emotions.

Group Allocation and Teaching: Using the breakout room feature in Zoom, participants were randomly assigned to either the experimental or control group. The teaching sessions for both groups were conducted simultaneously by two teachers, each employing their respective interaction modalities (MOI for the experimental group and limited interaction modalities for the control group).

Post-Instruction Emotion Assessment: Following the teaching session, participants in both groups received a second e-link to complete the BMIS for a second time.

2.6 Data analysis

In total, two surveys were conducted at two time points to assess changes in students' emotions: Time1 (before class and after watching the video clip) and Time2 (after class). The D-value (i.e., Time2 minus Time1) was calculated to compare changes in emotions between the two time points.

The BMIS results were analyzed using one-way multivariate analysis of variance (MANOVA) to determine if there were significant differences in mood items between the experimental and control groups based on the D-value.

Facial expression recognition was employed to reveal real-time dynamic emotional changes in participants during the teaching process. Emotional data obtained from participants' facial expressions were averaged and calculated for different parts of the teaching process to draw a line chart of basic emotional changes during learning. One-way MANOVA was used to determine the statistical significance of differences in emotions between different teaching parts and groups.

3 Results

3.1 Student's self-report emotions

Table 2 presents the results of a one-way MANOVA on 16 items in the BMIS, revealing significant differences between the experimental and control groups. Specifically, the experimental group demonstrated significantly higher levels of the positive emotions associated with online learning, Lively (F (1, 50) = 6.48, p = 0.014, partial η2 = 0.12) and Happy (F (1, 50) = 7.53, p = 0.08, partial η2 = 0.131). Conversely, there were significantly higher levels of the negative emotion Nervous (F (1, 50) = 6.43, p = 0.014, partial η2 = 0.11) and a lower level of Calm (F (1, 50) = 7.96, p = 0.007, partial η2 = 0.14) in the experimental group compared to the control group, indicating that online learners using MOI experienced not only positive emotions but also negative ones.Table 2 Results of one-way MANOVA on D-value in BMIS.

Table 2Mood Items	Experimental Group (N = 31)	Control Group (N = 27)	ANOVA	
M	SD	M	SD	F	P	η [2]	
Lively	0.52	0.57	0.11	0.58	6.48	0.014a	0.12	
Happy	0.58	0.62	0.11	0.51	7.53	0.008b	0.13	
Sad	−0.06	0.57	−0.15	0.77	0.06	0.800	0.00	
Tired	−0.10	0.79	−0.48	0.85	2.44	0.124	0.05	
Caring	0.16	0.58	0.26	0.59	0.33	0.568	0.01	
Content	0.00	0.89	0.22	0.85	1.37	0.247	0.03	
Gloomy	−0.19	0.75	0.00	0.62	1.41	0.240	0.03	
Jittery	−0.23	0.62	−0.11	0.58	0.89	0.349	0.02	
Drowsy	−0.26	0.68	−0.11	0.64	0.77	0.383	0.02	
Grouchy	−0.16	0.37	−0.19	0.56	0.17	0.686	0.00	
Peppy	0.23	0.76	0.11	0.97	0.42	0.518	0.01	
Nervous	0.39	0.84	−0.22	0.85	6.43	0.014a	0.11	
Calm	−0.32	0.79	0.15	0.60	7.96	0.007b	0.14	
Loving	0.23	0.72	0.19	0.74	0.08	0.781	0.00	
Fed up	−0.23	0.67	−0.11	0.42	0.84	0.365	0.02	
Active	0.45	0.85	0.26	0.66	0.52	0.473	0.01	
a p < 0.05.

b p < 0.01.

3.2 Facial expression recognition

During the teaching process, a total of 5129 images were collected from individual participants and analyzed using an automated tool to extract values of eight basic emotions, including Anger, Contempt, Disgust, Fear, Happiness, Neutral, Sadness, and Surprise.

As the teaching parts were not always synchronized and for the purpose of overcoming this issue. To enable a moment-to-moment comparison between two groups, we divided the course into seven parts based on the shared lesson plans and calculated the average emotional value of each participant at different teaching parts.

Finally, the average facial expression results for each teaching part were calculated separately for the experimental and control groups, and the trends are illustrated in Fig. 2. The results indicated that Neutral expressions accounted for the highest proportion of all emotions, with a recognition rate larger than 0.92 in both groups. To visualize the changing trends of the other emotions more effectively, we plotted the variation of Neutral using the range of 0.5–1 on the right Y-axis, while representing the remaining emotions (Anger, Contempt, Disgust, Fear, Happiness, Sadness, and Surprise) on the left Y-axis within the range of 0–0.2. the X-axis represents the different teaching phases over time, divided into seven parts: (1) Lead-in, (2) Vocabulary, (3) Text I, (4) Grammar & Practice I, (5) Text II, (6) Grammar & Practice II and (7) Summary & Culture.Fig. 2 Participants' real-time emotional changes based on expression recognition.

Fig. 2

Fig. 2 reveals that Neutral expressions dominated throughout all teaching parts in both the experimental and control groups, However, Neutral expressions cannot be simply interpreted as indicators of student engagement or disengagement, which will be explored in detail in the discussion section. In contrast, Anger, Disgust, Fear, and Contempt had consistently low average recognition rates, remaining below 0.01 with no significant variations during the course. Similarly, Surprise and Sadness exhibited low average recognition rates, falling below 0.025. Although minor peaks were observed in the control group's Surprise during Part 3 and the experimental group's Sadness during Part 6, statistical analysis via one-way MANOVA indicated no significant changes before and after these peaks.

Happiness, apart from Neutral expressions, displayed the highest average value among all emotions and demonstrated varying trends during the teaching process. Both the experimental and control groups began with moderate Happiness levels in Part 1, followed by a substantial increase in the final stage. However, starting from the middle segments, Happiness in the control group consistently declined. Conversely, the experimental group witnessed an increase in Happiness during Part 2 and Part 4, followed by a subsequent decrease. Particularly, the average Happiness level in Part 4 of the experimental group was significantly higher than that of the control group (F = 3.78; p = 0.029), while differences in Happiness among other segments were not statistically significant (p > 0.1). Correspondingly, Neutral emotions showed an opposite trend during these teaching phases.

4 Discussion

4.1 Effect of MOI on learners’ self-report emotions

The study investigated the impact of MOI on students' emotions in synchronous online Chinese L2 classes. Results from participants' self-reports using the BMIS showed that the experimental group reported significantly higher levels of Lively and Happy emotions compared to the control group, answering research question 1. This finding expands on the context established by previous research [27], suggesting that the intervention behavior of teachers can influence students' positive emotions not only in face-to-face classes but also in synchronous online environments.

The observed positive outcomes in this study could be attributed to the increased breadth of interaction facilitated by MOI, leading to enhanced quantity and quality of interaction. MOI enables teachers to initiate interactions with more potential students through texting or polling, resulting in a greater number of teacher-student interactions. This may address the issue of inadequate nonverbal communication with high breadth and low depth in online teaching, as highlighted by Harsch [15]. The heightened level of interaction leads to greater attention from teachers and increased behavioral engagement, ultimately fostering positive emotions in the online learning environment [6].

Additionally, MOI empowers teachers to provide precise, individualized feedback by efficiently assessing students' contributions. This streamlined approach allows teachers to evaluate responses effectively, improving knowledge acquisition and learner engagement [51]. Instructors can offer tailored feedback, enhancing online interaction quality and supporting personalized instruction [52]. This, in turn, contributes to the positive emotional states reported by students in the experimental group.

However, this study also found that MOI statistically significantly increased the level of Nervous and inhibited Calm among participants. This could be due to increased internal and external pressures. Firstly, MOI allows teachers to directly address questions to each student. As a result, students may need to respond more frequently than in the one-to-one interaction mode used in the control group.This can lead to a greater volume of information and an increased cognitive load for learners, potentially overwhelming their working memory [53]. Secondly, MOI often involved teachers asking questions to the entire class, with rapid feedback collection rather than waiting for all responses such a demand could potentially lead to language errors and further trigger anxiety and nervousness [24]. Lastly, teachers often share individual answers, albeit anonymously, after the poll, which may inflict invisible comparisons between students and increase their external pressures, hindering language learning [54].

To mitigate these challenges, we recommend that teachers applying MOI limit the overall number of questions directed to students to prevent potential burnout resulting from an overtaxing study load and high demands [55]. Additionally, compared to face-to-face teaching environments, increasing the waiting time after asking questions is a new requirement for teachers in online environments [8]. Finally, selectively displaying student answers may also be an effective way to reduce anxiety caused by academic competition [56].

In summary, the impact of MOI on students' self-reported emotions yields a mixed picture, encompassing both positive and negative effects. This finding support prior research [22,25] that suggests positive and negative emotions are not in a simple see-saw relationship. This study also extends the observations of Resnik and Dewaele, who noted that reduced interaction in online classrooms leads to diminished emotions among students [28]. However, the interaction mode in our study increases both the quantity and quality of interactions. This intensifies pressures and fosters greater emotional expression among learners in synchronous online classes.

4.2 Effect of MOI on emotions at a finer granularity

In addition to the student self-reported survey, this study also utilized facial expression recognition technology to provide a more nuanced perspective on the impact of MOI during online classes.

The dominant emotion observed was neutral, suggesting that attending these classes generally evokes neutral emotional responses. While neutral expressions can reveal students' emotional states, the relationship between these expressions and student engagement is complex and cannot be solely determined from facial expressions [57]. For example, Kinane et al. developed a video application that classified both happy and neutral expressions displayed by learners in synchronous online classes as states of engagement [58]. Furthermore, Nezami found that learners' facial expressions were mostly neutral during engagement [43]. Additionally, Dewaele and MacIntyre suggested that emotions may not be prevalent when an individual is highly focused [59]. However, other studies categorize neutral expressions as a state distinct from positive (enjoyment) and negative (confusion, fatigue, distraction, and neutral) academic emotions, representing a state where students do not exhibit any noticeable emotions, neither negative nor positive [60]. Therefore, it may be necessary to incorporate additional observable behavioral indicators, such as gaze, writing, and mimics, to more accurately assess students' engagement [61]. Given that this paper focuses on the impact of interaction modes on students' emotions, we will discuss the relationship between facial expressions and engagement in detail in a subsequent article.

Happiness emerged as the second most prevalent emotion expressed by students, indicating their enjoyment [60] and positive engagement [46]. Research also confirms that expressions of happiness expression are correlated with students' self-reported levels of understanding and satisfaction with the course [62]. Both experimental and control groups showed moderate happiness levels at the beginning, which increased significantly by the end. This aligns with Tonguç and Ozkara findings on students participating in an offline lecture using expression recognition technology [47]. However, the change in happiness observed in their research was not completely replicated in the present study. In their study, participants' happiness fluctuated slightly in the early stage and increased significantly in the middle and late stages of the lecture. These discrepancies may be attributed to differences in teaching content, language, and participant samples [37]. Tonguç and Ozkara did not report the specific teaching approaches used in the lecture [47], which might have affected participants' emotions. In light of the growing scholarly attention to the multimodal dimensions in the IRF sequence [18,19], facial expression recognition holds the potential not only to unveil positive emotions within the learning process but also to emerge as a new avenue for multimodal investigations in future online courses.

Our findings also support the impact of teacher-student interaction modes on students' emotional changes. A previous study recorded students' facial expressions while self-studying an introductory informatics course without teacher-student interaction, distinguishing expressions among six basic emotions (Disgusted, Surprised, Happy, Angry, Scared, Sad) and Neutral [63]. Consistent with our findings, Neutral expressions were the most frequent. However, in their study, the next most common emotions were Anger (20 %) and Sadness (9 %). In contrast, our study, which involved teacher-student interaction, revealed that Happiness was the second most prevalent emotion after Neutral, with very low proportions of Anger and Sadness. We propose that increased teacher-student interaction may have enhanced students' positive emotions [64].

10.13039/100014337 Furthermore , our results indicate that the impact of MOI on emotions varied across different teaching phases, further supporting this hypothesis. In Grammar & Practice Ⅰ, where MOI was frequently used, the experimental group reported significantly higher happiness levels than the control group. This could be attributed to the MOI's potential to promote a learner-centred approach by increasing the quantity and quality of the interaction between teacher and student. In an online environment, well-designed interactions have been shown to facilitate the transition from teacher-directed teaching to learner-centred teaching and contribute to the positive emotion [65].

However, in Grammar & Practice Ⅱ, which also applied MOI, the experimental group's happiness levels remained similar to the control group. Two possible reasons could explain this finding. First, it is possible that the participants' interest had decreased at that stage. This could be supported by Egbert's study, which found that the second iteration of a task elicited less positive emotional experience than the first time [66]. Second, it is likely that the students' attention had declined, as it has been suggested that adults' attention lasts for approximately 20 min [67]. Therefore, in teaching periods lasting longer than 20 min, the use of MOI may not significantly improve learners' happiness.

To sum up, based on the research question 2, the results of this study indicate that students' positive emotions underwent dynamic changes during the synchronous online class, as demonstrated by the facial expression recognition. Due to constraints posed by the research question and the scope of this article, a specific discussion regarding the correlation analysis between various self-reported measures and facial expression recognition results will be addressed in a separate article.

5 Conclusion

This study employed a quasi-experimental design to investigate the effects of Multiple Online Interaction (MOI) on students' emotions in synchronous online L2 classes. Participants' emotions were assessed using self-report measures and facial expression analysis. The results indicated that MOI had both positive and negative impacts on participants' emotions, including increased levels of lively and happy, alongside heightened nervous and reduced calm. Notably, facial expression recognition did not demonstrate a consistent improvement in emotional experiences throughout the class session; instead, the positive effects of MOI were primarily observed during the teaching segment focused on Grammar & Practice I, where MOI was most frequently utilized.

These findings highlight the advantages of MOI in broadening the breadth of interaction within L2 classes, thereby enhancing the quality of teacher-student interaction. MOI facilitates increased teacher outreach to a larger pool of students through text-based interactions or polling, fostering greater teacher-student exchanges that can boost attention and behavioral involvement. Moreover, MOI empowers teachers to deliver personalized feedback to individual students, encouraging additional one-on-one teaching opportunities.

Additionally, our research confirmed that facial expression recognition can serve as a valuable tool for measuring students' real-time emotions in the class. Teachers can use students' facial feedback to promptly adjust teaching pace, difficulty, and modes of interaction to achieve more personalized instruction. Future studies could further validate the relationship between facial expressions and student engagement and develop specialized software to support online teaching.

However, certain limitations should be acknowledged in this study. Firstly, the BMIS utilized in the assessment of emotions did not include certain emotions commonly experienced by L2 learners, such as boredom or confusion. Secondly, the study was conducted exclusively among university Chinese L2 students. Hence, the verification of these findings among learners of other languages requires more research. Finally, further research is needed to investigate the impact of long-term adoption of MOI as an interactive modality on students' positive and negative emotions.

Ethical approval, informed consent, and implied consent

This study and its experimental protocols were approved by the ethics committee of School of International Education in Shantou University (Project number:050121002; Date: 2021.9.30)., and all methods were conducted in compliance with the applicable guidelines and regulations. Informed consent was obtained from all participants prior to their participation in the study.

For the background survey questionnaire administered to participants before their involvement in the experiment, participants were initially asked to provide their ethical consent. Participants were informed about the experiment's details and were given the right to withdraw their data at any time. Completion of the questionnaire and participation in the experiment were considered as implicit consent to participate in the research.

Data availability

The data contains privacy-sensitive information related to the participants, particularly the expression data. Making the data publicly available might compromise confidentiality. However, we are willing to share the data with researchers who request it and agree to maintain the confidentiality of the participants.

CRediT authorship contribution statement

Xuhui Hu: Writing – review & editing, Writing – original draft, Software, Methodology, Investigation, Formal analysis, Data curation. Jian Gao: Writing – review & editing, Supervision, Project administration, Funding acquisition, Conceptualization. Daniela Romano: Writing – review & editing, Visualization, Validation, Supervision, Methodology, Formal analysis, Conceptualization.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Jian Gao reports financial support was provided by the Center for Language Education and Cooperation under grant number 21 YH25C. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

We wish to convey our sincere appreciation to Professor Jian-E PENG of Shantou University for her invaluable counsel and guidance during the development of this article. Furthermore, we extend our profound gratitude to the editors and anonymous reviewers of Heliyon for their invaluable insights and constructive feedback on the initial manuscript. Their meticulous review and thoughtful recommendations have greatly improved the quality and lucidity of our research.
==== Refs
References

1 Ellis R. The Study of Second Language Acquisition 1994 Oxford University
2 Kohnke L.M.A. Moorhouse B.L. Facilitating synchronous online language learning through Zoom RELC J. 53 2020 296 301
3 Ochs C. Gahrmann C. Sonderegger A. Learning in hybrid classes: the role of off-task activities Sci. Rep. 14 2024 1629 38238399
4 de Sá V.R. Anxiety and functional impairment affects undergraduate psychology students' learning in remote emergency teaching during the COVID-19 pandemic Sci. Rep. 13 2023 1503 36707652
5 Dewaele J.-M. Li C. Emotions in second language acquisition: a critical review and research agenda Foreign Lang. World 196 2020 34 49
6 Brooks C.F. Young S.L. Emotion in online college classrooms: examining the influence of perceived teacher communication behavior on students' emotional experiences Technol. Pedagog. Educ. 24 2015 515 527
7 Cheung A. Synchronous online teaching, a blessing or a curse? Insights from EFL primary students' interaction during online English lessons System 100 2021 102566
8 Hu X. A study of comparison of teacher questions of Chinese as a second language in synchronous online teaching and offline teaching and classroom time loss J. Yunnan Norm. Univ. Teach. Stud. Chin. Foreign Lang 19 2021 56 64
9 Martin F. Ahlgrim-Delzell L. Budhrani K. Systematic review of two decades (1995 to 2014) of research on synchronous online learning Am. J. Dist. Educ. 31 2017 3 19
10 Long M.H. The role of the linguistic environment in second language acquisition Ritchie W. Bhatia T. Handbook of Second Language Acquisition 1996 Academic Press San Diego, CA 413 468
11 Gass S.M. Varonis E.M. Input, interaction, and second language production Stud. Sec. Lang. Acquis. 16 1994 283 302
12 Cai S. Niu X. Wen Y. Li J. Interaction analysis of teachers and students in inquiry class learning based on augmented reality by iFIAS and LSA Interact. Learn. Environ. 2021 10.1080/10494820.2021.2012808
13 Rapanta C. Botturi L. Goodyear P. Guàrdia L. Koole M. Online university teaching during and after the COVID-19 crisis: refocusing teacher presence and learning activity Postdigital Sci. Educ. 2 2020 923 945
14 Alshaboul Y.M. The perceived challenges to online learning during the COVID-19 pandemic: a nationwide study of K-12 parental perspectives (Arab and other parents) in Qatar Heliyon 10 2024 e28578
15 Harsch C. Müller-Karabil A. Buchminskaia E. Addressing the challenges of interaction in online language courses System 103 2021 102673
16 Peng J. The roles of multimodal pedagogic effects and classroom environment in willingness to communicate in English System 82 2019 161 173
17 Carmi G. E-learning using zoom: a study of students' attitude and learning effectiveness in higher education Heliyon 2024 e30229 10.1016/j.heliyon.2024.e30229
18 Sindoni M.G. Mode-switching in video-mediated interaction: integrating linguistic phenomena into multimodal transcription tasks Ling. Educ. 62 2021 100738
19 Park Y. Park S. Eliciting student participation in synchronous online L2 lessons: the use of oral and written DIUs Ling. Educ. 71 2022 101085
20 Ekman P. An argument for basic emotions Cognit. Emot. 6 1992 169 200
21 Fredrickson B.L. Positive emotions broaden and build Advances in Experimental Social Psychology 2013 1 53 10.1016/b978-0-12-407236-7.00001-2
22 Li C. Dewaele J.-M. The predictive effects of trait emotional intelligence and online learning achievement perceptions on foreign language class boredom among Chinese university students Foreign Lang. Foreign Lang. Teach. 5 2020 33 44
23 Horwitz E.K. Horwitz M.B. Cope J. Foreign language classroom anxiety Mod. Lang. J. 70 1986 125 132
24 Yamada M. The role of social presence in learner-centered communicative language learning using synchronous computer-mediated communication: experimental study Comput. Educ. 52 2009 820 833
25 Dewaele J.-M. MacIntyre P.D. The two faces of Janus? Anxiety and enjoyment in the foreign language classroom Stud. Sec. Lang. Learn. Teach. 4 2014 237 274
26 Peng J. Wang Z. The predictive roles of enjoyment, anxiety, willingness to communicate on students’ performance in English public speaking classes Int. Rev. Appl. Linguist. Lang. Teach 62 2022 485 508
27 Alrabai F. The role of mixed emotions in language learning and teaching: a positive psychology teacher intervention System 107 2022 102821
28 Resnik P. Dewaele J.-M. Learner emotions, autonomy and trait emotional intelligence in ‘in-person’ versus emergency remote English foreign language teaching in Europe Appl. Ling. Rev. 2021 10.1515/applirev-2020-0096
29 Başal A. Eryılmaz A. Engagement and affection of pre-service teachers in online learning in the context of COVID 19: engagement-based instruction with web 2.0 technologies vs direct transmission instruction J. Educ. Teach. 47 2021 131 133
30 Tsang A. Dewaele J.M. The relationships between young FL learners’ classroom emotions (anxiety, boredom, & enjoyment), engagement, and FL proficiency Appl. Linguist. Rev. 2023 10.1515/applirev-2022-0077
31 Li C. Wei L. Anxiety, enjoyment, and boredom in language learning amongst junior secondary students in rural China: how do they contribute to L2 achievement? Stud. Sec. Lang. Acquis. 45 2023 93 108
32 Mayer J.D. Cavallaro R. Brief Mood Introspection Scale (BMIS): Technical and Scoring Manual 2019 University of New Hampshire
33 Mayer J.D. Gaschke Y.N. The experience and meta-experience of mood J. Pers. Soc. Psychol. 55 1988 102 111 3418484
34 Ferreri N. Mayhorn C.B. Individual differences in frustration and performance with online shopping activities Proc. Hum. Factors Ergon. Soc. Annu. Meet. 65 2021 148 153
35 Beedie C. Terry P.C. Lane A.N. Distinctions between emotion and mood Cognit. Emot. 19 2005 847 878
36 Bianchi-Berthouze N. Lisetti C.L. Modeling multimodal expression of user's affective subjective experience User Model. User-Adapted Interact. 12 2002 49 84
37 Li C. Dewaele J.-M. Jiang G. The complex relationship between classroom emotions and EFL achievement in China Appl. Ling. Rev. 11 2020 485 510
38 Krumpal I. Determinants of social desirability bias in sensitive surveys: a literature review Qual. Quantity 47 2013 2025 2047
39 Ekman P. Expressions and the nature of emotion Scherer K. Ekman P. Approaches to Emotion 1984 Lawrence Erlbaum 319 343
40 Melo C.M. Santos F.C. Terada K. Emotion expression and cooperation under collective risks iScience 26 2023
41 Lombardi G. Sciutti A. Rea F. Vannucci F. Di Cesare G. Humanoid facial expressions as a tool to study human behaviour Sci. Rep. 14 2024 133 38167552
42 Bartlett M.S. Hager J.C. Ekman P. Sejnowski T.J. Measuring facial expressions by computer image analysis Psychophysiology 36 1999 253 263 10194972
43 Nezami O.M. Automatic recognition of student engagement using deep learning and facial expression Lecture Notes in Computer Science 2020 273 289 10.1007/978-3-030-46133-1_17
44 Girard J.M. Cohn J.F. Mahoor M.H. Mavadati S.M. Rosenwald D.P. Social risk and depression: evidence from manual and automatic facial expression analysis 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG 2013 10.1109/fg.2013.6553748
45 Kutt K. Sobczyk P. Nalepa G.J. Evaluation of selected APIs for emotion recognition from facial expressions International Work-Conference On the Interplay between Natural And Artificial Computation 65–74 2022 Springer International Publishing Cham 10.1007/978-3-031-06527-9_7
46 Liu Y. Chen J. Zhang M. Rao C.D. Student engagement study based on multi-cue detection and recognition in an intelligent learning environment Multimed. Tool. Appl. 77 2018 28749 28775
47 Tonguç G. Ozkara B.O. Automatic recognition of student emotions from facial expressions during a lecture Comput. Educ. 148 2020 103797
48 Yu J. Huang C. Han Z. He T. Li M. Investigating the influence of interaction on learning persistence in online settings: moderation or mediation of academic emotions? Int. J. Environ. Res. Publ. Health 17 2020 2320
49 Kumar S. Gupta A. Facial expression recognition: a review Proceedings of the National Conference on Cloud Computing and Big Data 2015 Shanghai China 4 6
50 Marcusson-Clavertz D. Kjell O.N.E. Persson S. Cardeña E. Online validation of combined mood induction procedures PLoS One 14 2019 0217848
51 Sybing R. Examining dialogic opportunities in teacher-student interaction: an ethnographic observation of the language classroom Learn. Cult. Soc. Interact. 28 2021 100492
52 Blatchford P. Bassett P. Brown P. Examining the effect of class size on classroom engagement and teacher–pupil interaction: differences in relation to pupil prior attainment and primary vs. secondary schools Learn. InStruct. 21 2011 715 730
53 Chang T.-W. Kinshuk C.N.-S. Yu P.-T. The effects of presentation method and information density on visual search ability and working memory load Comput. Educ. 58 2012 721 731
54 MacIntyre P.D. Gregersen T. Emotions that facilitate language learning: the positive-broadening power of the imagination Stud. Sec. Lang. Learn. Teach. 2 2012 193 213
55 Salmela-Aro K. Read S. Study engagement and burnout profiles among Finnish higher education students Burn. Res. 7 2017 21 28
56 Posselt J.R. Lipson S.K. Competition, anxiety, and depression in the college classroom: variations by student identity and field of study J. Coll. Student Dev. 57 2016 973 989
57 Whitehill J. Serpell Z. Lin Y.C. Foster A. Movellan J.R. The faces of engagement: automatic recognition of student engagementfrom facial expressions IEEE Trans. Affect. Comput. 5 2014 86 98
58 Kinane Daouadji A. Bendella F. Improving E-learning by facial expression analysis Appl. Comput. Sci. 20 2024 126 137
59 Dewaele J.-M. MacIntyre P.D. El flujo en el aula de español como lengua extranjera (Flow in the Spanish foreign language classroom). In Factores cognitivos y afectivos en la enseñanza de español como lengua extranjera 2019 Routledge London 303 330
60 Bian C. Zhang Y. Yang F. Bi W. Lu W. Spontaneous facial expression database for academic emotion inference in online learning IET Comput. Vis. 13 2019 329 337
61 Zaletelj J. Košir A. Predicting students' attention in the classroom from Kinect facial and body features EURASIP J. Image Video Process 2017 1 12 10.1186/s13640-017-0228-8
62 Sathik M. Jonathan S.G. Effect of facial expressions on student's comprehension recognition in virtual educational environments SpringerPlus 2 2013 1 9 23419944
63 Terzis V. Moridis C.N. Economides A.A. Measuring instant emotions based on facial expressions during computer-based assessment Personal Ubiquitous Comput. 17 2013 43 52
64 Valiente C. Swanson J. DeLay D. Fraser A.M. Parker J.H. Emotion-related socialization in the classroom: considering the roles of teachers, peers, and the classroom context Dev. Psychol. 56 2020 578 32077726
65 Weinhandl R. Enhancing user-centred educational design: developing personas of mathematics school students Heliyon 10 2024 e24173
66 Egbert J. A study of flow theory in the foreign language classroom Can. Mod. Lang. Rev.-Rev. Can. Lang. Vivantes 60 2004 549 586
67 Cornish D. Dukette D. The Essential 20: Twenty Components of an Excellent Health Care Team 2009 Dorrance Publishing
