
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

39223215
71278
10.1038/s41598-024-71278-6
Article
Comparative analysis of artificial intelligence and expert assessments in detecting neonatal procedural pain
Giordano Vito vito.giordano@meduniwien.ac.at

1
Luister Alexandra 23
Vettorazzi Eik 4
Wonka Krista 1
Pointner Nadine 1
Steinbauer Philipp 1
Wagner Michael 1
Berger Angelika 1
Singer Dominique 2
Deindl Philipp 2
1 https://ror.org/05n3x4p02 grid.22937.3d 0000 0000 9259 8492 Department of Pediatrics and Adolescent Medicine, Division of Neonatology, Pediatric Intensive Care and Neuropediatrics, Comprehensive Center for Pediatrics, Medical University of Vienna, Waehringer Guertel 18-20, 1090 Vienna, Austria
2 grid.13648.38 0000 0001 2180 3484 Department of Neonatology and Pediatric Intensive Care Medicine, University Children’s Hospital, University Medical Center Hamburg Eppendorf, Hamburg, Germany
3 https://ror.org/01xnwqx93 grid.15090.3d 0000 0000 8786 803X Department of Neonatology and Pediatric Intensive Care Medicine, University Hospital Bonn, Bonn, Germany
4 https://ror.org/01zgy1s35 grid.13648.38 0000 0001 2180 3484 Institute of Medical Biometry and Epidemiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
2 9 2024
2 9 2024
2024
14 2037411 4 2024
26 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Assessing pain in newborns in the NICU is crucial due to their frequent exposure to painful stimuli, yet it's challenging due to the subjective nature of current methods. This study aimed to evaluate the effectiveness of an AI system designed for automatic facial recognition by comparing its performance with the expert opinion of health care provider. This is a secondary analysis from an eye-tracking study, assessing neonatal pain evaluations by healthcare professionals. The performance of AI software, FaceReader 9, was compared to experts' evaluations using a visual-analog scale, focusing on identifying specific facial action units associated with different pain levels. The study found significant differences in AI-generated metrics—arousal and valence—across three stimulus types: non-noxious thermal, short-noxious, and prolonged-noxious, with p-values below 0.001. A strong correlation (r = 0.84, p ≤ .001) was observed between AI metrics and expert ratings. Eleven facial action units were identified as relevant to describe neonatal pain. The findings highlight the AI system's potential in accurately detecting and analyzing newborn facial expressions in response to varying pain intensities, demonstrating a significant correlation with healthcare professionals' assessments. This suggests that AI technology could enhance objective pain assessment in neonates.

Subject terms

Neuroscience
Health care
Medical research
issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcThe identification of emotions is crucial for understanding both the evolutionary and developmental aspects of human behaviour, particularly in neonates where emotions are primarily innate and serve as a vital regulatory function1,2. The response of caregivers to a neonate's needs not only shapes their mutual relationship but also plays a significant role in the development of the child's self-regulatory capabilities3,4. In this early, preverbal stage of development, neonates primarily express emotions in response to physical stimuli, rather than through contextual or learned associations, making the caregiver’s role in interpreting these signals critical5.

Assessing pain in the Neonatal Intensive Care Unit (NICU) is vital, given the frequent exposure of newborns to painful procedures and stressors like separation, light, and noise6. Untreated pain adversely affects neonatal outcomes and reduces quality of life7, yet accurately assessing it remains challenging. Current methods lack objectivity, and biomarkers like cortisol are limited8. In neonatal care, particularly in Neonatal Intensive Care Units (NICUs), the challenge of interpreting neonatal emotions is compounded. Here, different healthcare providers must assess an infant's comfort and well-being, often in complex and high-stress environments9. Traditional methods, such as visual inspection of pain patterns and body movement, offer valuable insights but may not always align with the actual elicitors and context due to the less coordinated nature of neonatal responses10. This can lead to challenges in accurately interpreting neonatal expression9. In NICU settings, where infants are exposed to numerous stressful procedures and situations6,11, an accurate and objective assessment of distress and pain is essential. Currently, this assessment largely relies on patient observation and structured documentation using pain scales9, which, while necessary, can be subjective and prone to error due to the need for focused observation within a limited timeframe.

While some objective assessment tools exist, they're mainly used in research settings. Technological progress, particularly in machine learning and Artificial Intelligence (AI), offer promise for improving pain recognition and outcomes12. In recent years, the field of AI and automatic pattern emotion recognition has made substantial advancements13,14, particularly in the development of systems capable of real-time automatic face detection and emotion recognition in adults15–17. However, less progress has been made in understanding and applying these technologies to infants, especially those under six months of age17,18.

The objective of this study was to investigate the adaptability and efficacy of an AI system, initially developed for facial recognition in older children and adults, for assessing pain in neonates. We examined the AI tool’s capability to identify and quantify facial expressions in response to stimuli of varying intensities in newborn infants. We further aimed to evaluate the correlation between AI-derived metrics—arousal and valence—and pain evaluations conducted by experienced healthcare professionals.

Methods

Research environment

The study was conducted at the Level IV Perinatal center at University Hospital Hamburg-Eppendorf, Germany. AI-based emotion detection technology was applied to analyse the response of term neonates undergoing routine capillary blood sampling procedures intended for metabolic screening. The study protocol (PV7384) received ethical approval from the Ethics Committee of the Hamburg Medical Association. The research was performed in accordance with the Declaration of Helsinki following relevant guidelines. Informed consent was obtained from all participants (health-care-providers) and from all legal guardians of each neonates filmed during routine capillary blood sampling procedures.

Study framework

This research constitutes a secondary analysis of data initially gathered during an eye-tracking study, which was designed to assess neonatal pain evaluations conducted by healthcare professionals. The eye-tracking data was gathered using a Tobii Pro Fusion eye-tracker (2500 Hz, Tobii technology, Sweden) coupled with an LG Electronics monitor (24BK550Y, 1920 × 1080). Parental consent was obtained to film the newborns under three conditions: (1) Contact with a disinfectant-soaked swab (a non-noxious thermal stimulus), (2) A heel prick using an automatic lancet (SafetyLancet Neonatal, Sarstedt, Nümbrecht, Germany) (a short, noxious stimulus), and (3) Pressure applied to facilitate blood flow during capillary sampling (a prolonged noxious stimulus). The filming of the participating newborns, with prior parental consent, took place in a quiet room. The infants were positioned supine, and both video and audio were recorded—the camera (GoPro HERO 6, San Mateo, CA, USA) and microphone (iGoMic Mini Shotgun, MicW, China) were mounted on a tripod (DigiCharge Octopus 10″ Tripod, Digital Accessories Ltd, Derbyshire, UK) above the infant's torso level. As per local guidelines for nonpharmacological pain management in neonates, a 0.1 mL/kg dose of 30% glucose solution (DOLCEO, Chiesi GmbH, Hamburg) was administered one minute before all procedures, no rescue doses were further used.

Video analysis and pain assessment

The initial study involved presenting participants (health-care-providers) with 10-s sequences of 42 videos, each depicting 14 infants previously filmed in three distinct conditions: (1) a non-noxious thermal stimulus, (2) a short, noxious stimulus, and (3) a prolonged noxious stimulus. All infants were filmed at day two postpartum. The filming followed a precise timeline sequence according to the routine blood sampling procedure (Fig. 1A). The sequence began with the infant in a calm state, followed by the administration of sucrose as described. Using a GoPro (GoPro Hero10), fixed above the infant, we filmed the patient undisturbed. The recording continued by the application of the disinfectant (thermal stimulus), the heel prick, and finally the pressure on the heel to draw blood. Detailed information about the non-noxious thermal stimulus is provided in the supplementary material (Fig. S1). The procedures were carried out calmly with adequate support from the medical staff. There was no overlapping between the events having at least 30 s break between a stimulus and the other. Exact information about duration of the videos, pause between each condition can be found in supplementary material (Fig. S2). The videos were edited in post-processing to distinguish the different procedures and to fit into the duration of 10 s. Figure S2 clarify the inclusion of the stimulus within the selected 10 s. Seven infants were presented in a full-body view, while the other seven were shown in a face view only (see Supplementary Material). Post processing cut of the videos was performed by AL using (iMovie, Version 10.3.8, Apple, CA, USA) (see Supplementary material for detailed information regarding the post processing). Videos of the infants were then randomly presented using Tobii Pro Lab (version 1.194, released on June 7, 2022) to a representative sample of health-care-providers (participants) of the NICU. Participants were in part blinded to the context as video were presented in full body view (knowing the stimulus), or in face view only (blinded to the stimulus). After viewing each video, health-care-providers (participants) were asked to rate the infant's perceived pain using VAS (Fig. 1B). The videos were randomly displayed for evaluation to 46 healthcare providers. These providers assessed the videos using a Visual Analogue Scale (VAS) that ranged from 1 (indicating no pain) to 6 (indicating maximum pain), as depicted in Fig. S1. The VAS scale was chosen to align with the eye-tracking paradigm, enabling healthcare professionals to assess infants' pain using the same technology (Fig. 1B).Figure 1 Representation of the procedure in the study. Figure (A) indicates the filming sequence. Figure (B), on the other hand, shows the eye-tracking experiment in which the participants (health-care providers) evaluated the videos, with the red bar indicating the eye tracker.

FaceReader analysis

For the purposes of the current research, the videos underwent re-analysis employing the AI-based FaceReader software (Version 9, Noldus, Amsterdam, Netherlands), using baby face model. FaceReader is a facial analysis software capable of detecting facial expressions. The software, previously trained on adults (General Face model) and infants up to six months old (Baby Face model), had not been used on newborns prior to this study. The process involves three key steps: face finding, face modelling, and face classification. Face finding employs a deep learning algorithm to identify potential facial regions in an image at various scales. Face modelling uses a facial modelling technique based on deep neural networks to create an artificial face model that locates 468 key points on the face, providing a comprehensive set of facial landmarks18. For face classification, neural networks analyse the image pixels to categorize facial expressions, drawing on a database of several manually coded images14,17,18. FaceReader also provides detailed information on individual facial action units as per the Facial Action Coding System (FACS) developed by Ekman and colleagues18,19 and it´s derivate BabyFACS, as well as data on the activation level of each facial expression. The FaceReader is thought to detect following emotion in adult: happy, sad, angry, surprised, scared, disgusted, and neutral. It also provides the opportunity to customize groups of action units, working therefore with extreme flexibility. The FaceReader, further provides previous mentioned information on continuum where the valence indicate the emotional polarity of the expression, while the arousal its intensity level. In this specific case, using the module baby cry of FaceReader 9, valence was represented on a scale between 0 and 1 where higher values corresponded to higher level of pain while scores closed to 0 a more neutral face expression.

Statistical analysis

The analysis was conducted using R Studio (provided by the R Foundation for Statistical Computing, Vienna, Austria) and JASP (Version 0.18.3, JASP Team, 2024; University of Amsterdam). The facial action units (AUs) were depicted using barographs and boxplots for each category of stimulus: (1) non-noxious thermal stimulus, (2) short noxious stimulus, and (3) prolonged noxious stimulus. To visualize and analyze the effects of these stimuli on the AI-generated metrics arousal and valence, line graphs were employed.

Continuous variables are presented as means ± standard deviations (SD), while categorical variables are shown as counts and percentages. Pearson correlation analysis was applied to explore the relationship between arousal, valence, and expert assessments. An Analysis of Variance (ANOVA) was utilized to determine differences in AU activation across the three stimulus conditions, with the Tukey test serving as a post-hoc analysis for multiple comparisons. F-values and η2 were reported to estimate effect size. A Multivariate Analysis of Variance (MANOVA) was performed to examine the differences in selected AUs across conditions, employing Pillai's trace test to assess the impact of contributing factors on the model. A Delta was calculated to understand the difference between the minimum and maximum activity levels of each AU. An AU was deemed relevant if it met two key criteria: a mean activity level of at least 15% and a minimum delta of 50%. These thresholds were set to ensure that only AUs demonstrating both notable mean activity and significant dynamic variability were selected for further analysis. Lastly a net plot was drawn to understand correlation in a dynamic way between relevant AU previously identified.

Results

Participant demographics and pain assessment

Descriptive characteristics of the infants are presented in Table 1. All infants were filmed at day two. The C-section ratio in our cohort was higher compared to the general ratio of C-sections documented at the UKE, which is reported to be 32.6%. This discrepancy is likely because researchers had a higher chance of recruiting parents of children for whom a C-section was planned, providing more time to adequately speak with the parents and organize the filming of the situation. In the initial study, the pain responses of newborns were evaluated by 46 healthcare professionals, comprising 29 nurses (63%) and 17 physicians (37%). Within this group, the gender distribution among physicians was 53% female, while the nursing staff was exclusively female. The range of professional experience among the participants varied, with 34.8% having 0–5 years of experience, 26.1% having 6–10 years, 8.7% having 11–15 years, 6.5% having 16–20 years, and 23.9% having more than 20 years of experience.Table 1 Infants descriptive characteristics.

Variables	Mean	Std. Deviation	Minimum	Maximum	25th percentile	50th percentile	75th percentile	
Gestational week	38.57	1.39	36.00	41.00	38.00	38.00	39.00	
Birth weight-g	3416.14	509.00	2574.00	4380.00	3184.50	3470.00	3717.50	
Apgar 5 min	9.71	0.46	9.00	10.00	9.25	10.00	10.00	
Head circumference-cm	35.02	1.23	33.00	37.50	34.12	35.00	35.95	
Length-cm	52.14	2.53	47.00	56.00	51.00	52.50	53.75	
Length of stay-days	2.71	0.99	2.00	5.00	2.00	2.00	3.00	
Film day	2	0	2	2	2	2	2	
Delivery modus (c-section) n,%	8 (n)	57%						
sex (Female) n,%	11 (n)	78.5%						

Stimulus intensity levels

The mean (SD) VAS scores were 2.3 (0.3) for the brief non-painful thermal stimulus, 4.1 (0.5) for the brief noxious stimulus, and 5.0 (0.3) for the prolonged noxious stimulus. Significant variations in total mean differences were observed across the three conditions for both AI generated metrics: arousal (p < 0.001; F = 17.35; η2 = 0.49) and valence (p < 0.001; F = 13.10; η2 = 0.42). Post hoc analyses indicated distinctions between the non-noxious stimulus compared to both the short noxious stimulus and the prolonged noxious stimulus. However, no significant differences were detected between the short and prolonged noxious stimuli in terms of both arousal and valence activations as recorded by the AI software (refer to Table 2).Table 2 Post hoc comparisons.

Arousal	Mean difference	SE	ptukey	
(Non-noxious thermal stimulus)	Short noxious stimulus	−0.302	0.070	 < .001	
Prolonged noxious stimulus	−0.391	0.069	 < .001	
Short noxious stimulus	Prolonged noxious stimulus	−0.088	0.067	0.398	
Post hoc comparisons	
Valence	Mean difference	SE	ptukey	
(Non-noxious thermal stimulus)	Short noxious stimulus	−0.213	0.063	0.005	
Prolonged noxious stimulus	−0.310	0.061	 < .001	
Short noxious stimulus	Prolonged noxious stimulus	−0.098	0.060	0.249	

In the analysis of video recordings, precise temporal markers were established to denote the instances of stimulus application, enabling the AI software to correlate these moments with observed shifts in arousal and valence. Figure 2 shows a graphical representation of the AI metrics Arousal and Valence timelines for all individual infants (thin lines) and the three stimuli (Fig. 2A) and an average for both AI metrics (thick red lines). We found distinct response patterns to non-noxious thermal stimulation, exhibiting significant variability across subjects, indicating a degree of individual subjectivity in the perception of this stimulus. Notably, both the short and prolonged noxious stimuli elicited detectable deviations from baseline measurements. The non-noxious thermal stimulus exhibited low levels of both arousal and valence throughout the duration, with a minor decline observed towards the end. In contrast, the short noxious stimulus demonstrated a progressive increase in activation from 4 to 8 s, peaking at approximately 8 s and then stabilizing for the remainder of the event. The prolonged noxious stimulus, meanwhile, started with high levels of arousal and valence from the onset and maintained this intensity relatively unchanged throughout the event (Fig. 2A). The middle part of the Fig. 2B illustrates the precise relative start and end times for each participant in relation to the stimuli. The lower part of the Fig. 2C displays the count of data points included in the calculation of the average AI metrics. Mean values were calculated only when data from more than five infants were available at the respective time points.Figure 2 The upper part (A) displays the raw data sets of the AI metrics arousal and valence for all infants in relation to the three stimuli non-painful thermal, short noxious, and prolonged noxious stimulus (indicated by the vertical dotted black line), with infants color-coded. The thick red lines represent the AI metric average. The middle part (B) illustrates the precise relative start and end times for each infant in relation to the stimuli. The lower part (C) displays the count of data points included in the calculation of the average AI metrics. In some cases, data was missing due to failed face detection caused by movements and suboptimal angles of the face, resulting in incomplete timelines.

Association between AI-generated metrics and expert pain assessments

The average VAS scores, as evaluated by 46 experts, were recorded as follows: 2.3 (± 1.2) for the non-noxious thermal stimulus, 4.1 (± 1.1) for the short noxious stimulus, and 5.0 (± 1.0) for the prolonged noxious stimulus. Analysis revealed significant variances in mean pain assessments across the three stimuli conditions (p < 0.001; F = 53.59; η2 = 0.74), indicating a clear differentiation in perceived pain levels. Additionally, the ratings of the stimuli exhibited a strong linear correlation with the AI-generated metrics of arousal and valence, as illustrated in Fig. 3. Further analysis revealed a strong Pearson correlation between arousal and expert pain assessments, with a correlation coefficient r = 0.84 (p =  < 0.001), and a similarly strong correlation for valence, with r = 0.86 (p =  < 0.001). These findings indicate a significant positive linear relationship between both AI-generated metrics and expert opinions on pain intensity.Figure 3 Correlation between human pain assessments by healthcare professionals (n = 46) using the Visual Analogue Scale (VAS) and the AI-generated metrics for all patients (n = 14) in different conditions (n = 42 videos): arousal (black regression line) and valence (dot size).The average VAS score of each participant was plotted against the AI level of activation.

Additionally, the type of stimulus showed a robust positive correlation with expert ratings r = 0.84 (p =  < 0.001), suggesting that the nature of the stimulus closely aligns with the experts' pain evaluations. The analysis also uncovered significant correlations between the type of stimulus and the AI-generated metrics, with arousal exhibiting a correlation coefficient of r = 0.66 (p =  < 0.001), and valence showing r = 0.63 (p =  < 0.001).

Identification of key facial action units in neonatal pain detection

By the graphical representation of AUs (Fig. 4), different activation profiles could be observed following different stimuli. In addition, a rough estimate could be made of which AUs were active on average and which showed a strong dynamics or differences between the individual stimuli.Figure 4 Box-plot representation of all considered action units (AUs) and their levels of activation across the three different stimuli: non-painful thermal, short noxious, and prolonged noxious stimuli (n = 42 videos).

We calculated the delta, representing the difference between the minimum and maximum activity levels of each AU, in addition to their mean activity across both short and prolonged noxious stimuli scenarios. An AU was deemed relevant if it met two key criteria: a mean activity level of at least 15% and a minimum delta of 50%. These thresholds were set to ensure that only AUs demonstrating both notable mean activity and significant dynamic variability were selected for further analysis. This method resulted in the selection of 11 (61%) of 18 AUs: 03, 04, 06, 07, 09, 10, 17, 20, 25, 26, and 27. To visually depict this selection process, we plotted the mean activity and delta for each AU in Fig. 5.Figure 5 Mean change (Delta) and mean activity level (Mean) of the examined Action Units (AUs). The dashed red lines indicate the defined thresholds for the relevance of the AUs.

Temporal visualization of neonatal facial responses to stimuli

We aimed to comprehensively analyze how neonates' facial expressions change in response to different types of stimuli. We therefore visualized the dynamic facial responses systematically according to stimulus and facial region on a timeline, as presented in Fig. 6. This approach allowed for an in-depth examination of the temporal patterns of AU activity, offering valuable insights into the nuanced facial expressions of neonates in reaction to various stimuli over time.Figure 6 Dynamic responses of the selected Action Units (AUs) over time for the three stimuli, categorized according to the facial area.

Our focused analysis on selected relevant AUs revealed distinct changes in neonatal facial expressions during exposure to different stimuli. Notably, exposure to the short noxious stimulus resulted in a pronounced increase in activity across these AUs (mean = 39%, delta = 92%), indicating a clear and significant reaction to this type of stimulation. Conversely, the non-noxious thermal stimulus elicited smaller changes in facial expression over time, characterized by a generally low level of AU activity (mean = 24%, delta = 86%). This contrasts sharply with the response to the prolonged noxious stimulus, which elicited a sustained high level of activity in the AUs, reflecting a marked and continuous facial expression of distress (mean = 50%, delta = 86%).

Finally, the interaction of different action units and their dynamic correlation pattern are reported in Fig. 7.Figure 7 Interaction of different Action Units and their correlation patterns illustrated as a netplot.

Discussion

In this study, we utilized the AI-based FaceReader 9 software to examine facial responses in newborns to various stimuli, correlating these responses with assessments from 46 healthcare professionals using a visual analogue scale. Our findings highlight a strong correlation between AI-detected facial action units and expert opinions on neonatal pain, indicating the potential of AI tools in enhancing pain assessment precision.

Correlation between AI metrics arousal and valence, and clinical pain assessments

Research in automatic facial pattern detection has seen significant advancements since its inception in 200612, with recent developments favoring sophisticated AI models over traditional machine learning approaches20. Despite progress, the application of these technologies to infants, especially those under two months old, remains limited12. Unlike most algorithms that rely on static images, our study leverages video analysis to closely mimic real-life scenarios, providing a more accurate reflection of neonatal responses to pain and discomfort. Our findings reveal a direct correlation between the neonates’ pain response as rated by healthcare professionals and the arousal levels detected by AI, underscoring the potential of AI tools in clinical settings. This study stands out by comparing AI metrics with assessments from a broad expert panel, focusing on the critical age group of infants under two months. Hoti et al.'s recent work in Lancet Digital Health21 validated the PainCheck tool's reliability in video assessments of infant pain, further supporting the utility of AI in pain detection. Similarly, research by Hammal and colleagues22, as well as Braham et al.12, highlights the effectiveness of AI in distinguishing between different emotional states in infants, emphasizing the diverse potential applications of these technologies in pediatric care.

Specific face action units to identify painful responses in neonates

For the subset of patients where precise stimulus timing was achievable, there was a notable and distinct surge in both arousal and valence, especially in response to the short noxious stimulus. This indicates the AI’s capability to precisely track and reflect time changes in facial expressions. Such findings suggest the AI's potential for distinguishing between baseline and stimulus-induced expressions in future research.

Moreover, the software was able to detect the combination of relevant muscle activities related to the intensity of cry in newborn. Particularly, when the stimulus was a prolonged noxious stimulus an accentuated facial muscular movement was recorded. Most of the activities in this case concerned region of muscles close to the eye, knitting and knotting around the brow, raising the cheek and stretching the lip. By the visual inspection of AU different activation profiles could be observed in baby cry. However, when looking closely at the activation power of each AU only the following remained relevant: 3 and 4 (brow knitting and knotting); 6 (cheek raiser); 7 (lid tightener); 9 (nose wrinkle); 10 (lip raiser); 17 (chin raiser); 20 (lip stretcher); 25 (lips part); 26 (jaw drops); 27 (mouth stretch). Our results are partially in line with what has been described in the previous literature23. According to Hamal and colleagues23, AU 20 is one of the most relevant identifying cry followed by other region of interest: (AUs: 1, 2, 3, 4, 6, 9, 12, 20, 28). Our results are further partially in line with face physical distress described by the neonatal Facial coding system (NFCS) including: forehead protusion, contraction of the eyelids, horizontal stretch of the mouth, nasonabial groove, tense tongue24. Divergence with previous literature could be related to the different technology used and its application to a dataset of videos investigating pain reaction in a younger collective.

Previous studies have also examined the intensity of pain expression in neonates across key facial areas25. These studies revealed that, despite individual differences in timing and intensity, overall facial movements in response to painful stimuli are remarkably similar in newborns, regardless of their sex or racial/ethnic background25,26. This led to the introduction of the term “primal face of pain” (PFP), describing a basic, inborn expression associated with the unmistakable communication of distress essential for species survival26. In neonates, emotional facial expressions are largely influenced by genetic mechanisms, particularly during the first days of life due to limited social learning. However, there are distinct individual variations in the timing and intensity of facial pain expressions among infants25.

Current pain scales that focus of facial expression do not accurately reflect the natural intervals and progression of the PFP25. Introducing AI tools capable of integrating more anatomically faithful intervals into a given timeline could significantly improve pain assessment in the neonatal population.

The investigated technology (FaceReader) has already demonstrated validity in detecting different emotions in children older than six months and in adults 20. This seems to be reasonable as more complex emotions start to develop around this age 27–29. Nonetheless, more basic emotion related to discomfort or distress are already present at birth and expressed mostly through cry 30. Therefore, its automatic identification and the classification of its level could be of particular relevance in a NICU setting. Its functionality in newborn compared to older children or adults is not self-assured as facial features evolve rapidly over the life-span 12. Textural differences between infant and adults relay for example on skin ageing, face muscle elasticity, and face motion dynamics 12.

Limitations

Despite the valuable insights gained, this study has certain limitations. Firstly, due to the need to conduct the eye-tracking experiments within a reasonable timeframe for healthcare professionals, only a small, randomly selected sample of infant videos was presented. Additionally, personal biases of the healthcare professionals and cultural differences were not accounted. Furthermore, the VAS scale was used to fit the eye-tracking paradigm, reflecting the expert opinions of healthcare providers. While VAS scales are straightforward and easy to use for quick assessments, they may reflect more subjective judgments of pain31. Discordance opinion on its psychometrics properties have been reported alongside the literature despite evidence of sensitivity and responsivity to pain32,33. Moreover, and most importantly, the software was not tested in a live setting but only using videos of infants without pathologies or morbidities. Its accuracy in a clinical setting needs further investigation, particularly with patients who have different pathologies. The proprietary nature of the AI tool's algorithm for calculating arousal and valence metrics restricts our study to presenting descriptive correlations with clinical observations, without delving into the algorithm's operational specifics. It is also important to note that the algorithm's performance may be compromised by variations in angle and lighting conditions. In determining the relevant AUs, our study employed a straightforward selection criterion focused on mean activity and maximum activity change. While this approach ensures clarity and rigor, it may not fully capture the nuanced complexity inherent in facial expression analysis. Our study exclusively analyzed term born neonates, implying that their responses may not be directly applicable to preterm infants or those receiving care in Neonatal Intensive Care Units. Finally, between the heel prick and the pressure on the heel, the timing adhered to the procedure's duration. Therefore, even though a pause was set between the events and an increased painful reaction was observed by the squeezing of the heel, establishing a clear baseline was not possible given the consecutive nature of the events (see Supplementary Material for detailed information regarding the post processing of the videos).

Conclusion

This investigation sheds light on the potential of AI technology to accurately detect and analyze facial expressions in newborns in response to stimuli of varying intensities. It successfully demonstrates a significant correlation between AI-generated metrics—arousal and valence—and the pain evaluations conducted by healthcare professionals using a visual analogue scale. Moreover, the study identified crucial Facial Action Units that play a role in neonatal pain expressions. These results lay the groundwork for future applications of AI in healthcare, particularly in recognizing and assessing discomfort in newborns within real-world settings, marking an important step towards the integration of AI in enhancing patient care.

Supplementary Information

Supplementary Information.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-024-71278-6.

Acknowledgements

We would like to thank Marian Bittner & Dr. Amogh Gudi for their technical assistance.

Author contributions

Dr. Deindl, and Dr. Giordano conceptualized and designed the study. Dr. Deindl and Dr. Giordano coordinated and supervised data collection, performed statistics, wrote the first draft of the manuscript, and reviewed and revised the manuscript. Dr. Giordano programmed the eye-tracking experiment and measured participants. Dr. Luister helped collect the videos, collected data, and revised and widened the initial manuscript for important intellectual content. MSc. Pointner, Mag. Wonka, and Dr. Steinbauer helped in the eye-tracking paradigm and coding of the videos, preparation of tables, revised the manuscript, and approved its final version. Dr. Vettorazzi supervised statistical analyses. Prof. Berger, Dr. Wagner, and Prof. Singer advised on the study design and methods. In addition, they helped interpret the data and critically revised the draft for important intellectual content.

Data availability

All data generated or analyzed during this study are included in this published article (and its supplementary information files). The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Competing interests

The authors declare no competing interests.

Ethical approval

The study protocol (PV7384) received ethical approval from the Ethics Committee of the Hamburg Medical Association. The research was performed in accordance with the Declaration of Helsinki following relevant guidelines.

Informed consent

Informed consent was obtained from all participants (health-care-providers) and from all legal guardians of each neonates filmed during routine capillary blood sampling procedures.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Rodríguez-Torres R Leyens JP Pérez AR Rodriguez VB Quiles del Castillo MN Demoulin S The lay distinction between primary and secondary emotions: A spontaneous categorization Int. J. Psychol. 2005 40 2 100 107 10.1080/00207590444000221
Rodríguez-Torres, R. et al. The lay distinction between primary and secondary emotions: A spontaneous categorization. Int. J. Psychol. 40(2), 100–107 (2005).10.1080/00207590444000221
2. Tracy JL Randles D Four models of basic emotions: A review of Ekman and Cordaro, Izard, Levenson, and Panksepp and Watt Emot. Rev. 2011 3 4 397 405 10.1177/1754073911410747
Tracy, J. L. & Randles, D. Four models of basic emotions: A review of Ekman and Cordaro, Izard, Levenson, and Panksepp and Watt. Emot. Rev. 3(4), 397–405 (2011).10.1177/1754073911410747
3. Bowlby J May DS Solomon M Attachment Theory 1989 Lifespan Learning Institute
Bowlby, J., May, D. S. & Solomon, M. Attachment Theory (Lifespan Learning Institute, 1989).
4. Clark CA Woodward LJ Horwood LJ Moor S Development of emotional and behavioral regulation in children born extremely preterm and very preterm: biological and social influences Child Dev. 2008 79 5 1444 1462 10.1111/j.1467-8624.2008.01198.x 18826535
Clark, C. A., Woodward, L. J., Horwood, L. J. & Moor, S. Development of emotional and behavioral regulation in children born extremely preterm and very preterm: biological and social influences. Child Dev. 79(5), 1444–1462 (2008).18826535 10.1111/j.1467-8624.2008.01198.x
5. Ainsworth MDS Bell SM Stayton DF Infant-mother attachment and social development: Socialization as a product of reciprocal responsiveness to signals The integration of a child into a social world 1974 Cambridge University Press 99 135
Ainsworth, M. D. S., Bell, S. M. & Stayton, D. F. Infant-mother attachment and social development: Socialization as a product of reciprocal responsiveness to signals. In The integration of a child into a social world 99–135 (Cambridge University Press, 1974).
6. Carbajal R Rousset A Danan C Coquery S Nolent P Ducrocq S Epidemiology and treatment of painful procedures in neonates in intensive care units Jama 2008 300 1 60 70 10.1001/jama.300.1.60 18594041
Carbajal, R. et al. Epidemiology and treatment of painful procedures in neonates in intensive care units. Jama 300(1), 60–70 (2008).18594041 10.1001/jama.300.1.60
7. Giordano V Deindl P Gal E Unterasinger L Fuiko R Steinbauer P Pain and neurodevelopmental outcomes of infants born very preterm Dev. Med. Child Neurol. 2023 65 8 1043 1052 10.1111/dmcn.15505 36647629
Giordano, V. et al. Pain and neurodevelopmental outcomes of infants born very preterm. Dev. Med. Child Neurol. 65(8), 1043–1052 (2023).36647629 10.1111/dmcn.15505
8. Ng PC Is there a “normal” range of serum cortisol concentration for preterm infants? Pediatrics 2008 122 4 873 875 10.1542/peds.2008-0516 18829815
Ng, P. C. Is there a “normal” range of serum cortisol concentration for preterm infants?. Pediatrics 122(4), 873–875 (2008).18829815 10.1542/peds.2008-0516
9. Giordano V Edobor J Deindl P Wildner B Goeral K Steinbauer P Werther T Berger A Olischar M Pain and sedation scales for neonatal and pediatric patients in a preverbal stage of development: A systematic review JAMA Pediatr. 2019 173 12 1186 1197 10.1001/jamapediatrics.2019.3351 31609437
Giordano, V. et al. Pain and sedation scales for neonatal and pediatric patients in a preverbal stage of development: A systematic review. JAMA Pediatr. 173(12), 1186–1197 (2019).31609437 10.1001/jamapediatrics.2019.3351
10. Fitzgerald M The development of nociceptive circuits Nat. Rev. Neurosci. 2005 6 7 507 520 10.1038/nrn1701 15995722
Fitzgerald, M. The development of nociceptive circuits. Nat. Rev. Neurosci. 6(7), 507–520 (2005).15995722 10.1038/nrn1701
11. Bertsch M Reuter C Czedik-Eysenberg I Berger A Olischar M Bartha-Doering L The “Sound of Silence” in a neonatal intensive care unit-listening to speech and music inside an incubator Front. Psychol. 2020 11 1055 10.3389/fpsyg.2020.01055 32528386
Bertsch, M. et al. The “Sound of Silence” in a neonatal intensive care unit-listening to speech and music inside an incubator. Front. Psychol. 11, 1055 (2020).32528386 10.3389/fpsyg.2020.01055
12. Hassan T Seus D Wollenberg J Weitz K Kunz M Lautenbacher S Automatic detection of pain from facial expressions: A survey IEEE Trans. Pattern Anal. Mach. Intell. 2021 43 6 1815 1831 10.1109/TPAMI.2019.2958341 31825861
Hassan, T. et al. Automatic detection of pain from facial expressions: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(6), 1815–1831 (2021).31825861 10.1109/TPAMI.2019.2958341
13. Bartlett, M. S., Littlewort, G., Fasel, I., Movellan, J. R. (eds). real time face detection and facial expression recognition: development and applications to human computer interaction. In 2003 Conference on Computer Vision and Pattern Recognition Workshop; 2003 16–22 June 2003.
14. Bulat, A., Tzimiropoulos, G. How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230, 000 3D Facial Landmarks). 2017. p. 1021–30.
15. Loy CC Luo P Huang C Feris RS Lampert C Parikh D Deep learning face attributes for detection and alignment Visual Attributes 2017 Springer International Publishing 181 214
Loy, C. C., Luo, P. & Huang, C. Deep learning face attributes for detection and alignment. In Visual Attributes (eds Feris, R. S. et al.) 181–214 (Springer International Publishing, 2017).
16. Savran, A., Sankur, B., (eds). Automatic detection of facial actions from 3D data. In 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops; 2009 27 Sept.-4 Oct. 2009.
17. Zafeiriou S Zhang C Zhang Z A survey on face detection in the wild: Past, present and future Comput. Vis. Image Underst. 2015 138 1 24 10.1016/j.cviu.2015.03.015
Zafeiriou, S., Zhang, C. & Zhang, Z. A survey on face detection in the wild: Past, present and future. Comput. Vis. Image Underst. 138, 1–24 (2015).10.1016/j.cviu.2015.03.015
18. Gudi, A., Tasli, H. E., Den Uyl T. M., Maroulis, A. (eds.). Deep learning based FACS Action Unit occurrence and intensity estimation. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG); 2015 4–8 May 2015.
19. Ekman P An argument for basic emotions Cogn. Emot. 1992 6 3–4 169 200 10.1080/02699939208411068
Ekman, P. An argument for basic emotions. Cogn. Emot. 6(3–4), 169–200 (1992).10.1080/02699939208411068
20. Onal Ertugrul I Ahn YA Bilalpur M Messinger DS Speltz ML Cohn JF Infant AFAR: Automated facial action recognition in infants Behav. Res. Methods 2023 55 3 1024 1035 10.3758/s13428-022-01863-y 35538295
Onal Ertugrul, I. et al. Infant AFAR: Automated facial action recognition in infants. Behav. Res. Methods 55(3), 1024–1035 (2023).35538295 10.3758/s13428-022-01863-y
21. Hoti K Chivers PT Hughes JD Assessing procedural pain in infants: a feasibility study evaluating a point-of-care mobile solution based on automated facial analysis Lancet Digit. Health 2021 3 10 e623 e634 10.1016/S2589-7500(21)00129-1 34481769
Hoti, K., Chivers, P. T. & Hughes, J. D. Assessing procedural pain in infants: a feasibility study evaluating a point-of-care mobile solution based on automated facial analysis. Lancet Digit. Health 3(10), e623–e634 (2021).34481769 10.1016/S2589-7500(21)00129-1
22. Hammal Z Cohn JF Heike C Speltz ML Automatic measurement of head and facial movement for analysis and detection of infants positive and negative affect Front. ICT 2015 2 21 10.3389/fict.2015.00021
Hammal, Z., Cohn, J. F., Heike, C. & Speltz, M. L. Automatic measurement of head and facial movement for analysis and detection of infants positive and negative affect. Front. ICT 2, 21 (2015).10.3389/fict.2015.00021
23. Hammal, Z., Chu, W. S., Cohn, J. F., Heike, C., Speltz, M. L. (eds). Automatic action unit detection in infants using convolutional neural network. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) 2017 23–26 Oct. 2017.
24. Martínez A Pujol FA Mora H Application of texture descriptors to facial emotion recognition in infants Appl. Sci. 2020 10 3 1115 10.3390/app10031115
Martínez, A., Pujol, F. A. & Mora, H. Application of texture descriptors to facial emotion recognition in infants. Appl. Sci. 10(3), 1115 (2020).10.3390/app10031115
25. Schiavenato M von Baeyer CL A quantitative examination of extreme facial pain expression in neonates: The primal face of pain across time Pain Res. Treat. 2012 2012 251625 22645679
Schiavenato, M. & von Baeyer, C. L. A quantitative examination of extreme facial pain expression in neonates: The primal face of pain across time. Pain Res. Treat. 2012, 251625 (2012).22645679
26. Schiavenato M Facial expression and pain assessment in the pediatric patient: The primal face of pain J. Spec. Pediatr. Nurs. 2008 13 2 89 97 10.1111/j.1744-6155.2008.00140.x 18366376
Schiavenato, M. Facial expression and pain assessment in the pediatric patient: The primal face of pain. J. Spec. Pediatr. Nurs. 13(2), 89–97 (2008).18366376 10.1111/j.1744-6155.2008.00140.x
27. Holodynski M The internalization theory of emotions: A cultural historical approach to the development of emotions Mind Cult. Act. 2013 20 1 4 38 10.1080/10749039.2012.745571
Holodynski, M. The internalization theory of emotions: A cultural historical approach to the development of emotions. Mind Cult. Act. 20(1), 4–38 (2013).10.1080/10749039.2012.745571
28. Saarimäki H Gotsopoulos A Jääskeläinen IP Lampinen J Vuilleumier P Hari R Discrete neural signatures of basic emotions Cereb. Cortex 2015 26 6 2563 2573 10.1093/cercor/bhv086 25924952
Saarimäki, H. et al. Discrete neural signatures of basic emotions. Cereb. Cortex 26(6), 2563–2573 (2015).25924952 10.1093/cercor/bhv086
29. Yik M Russell JA Steiger JH A 12-point circumplex structure of core affect Emotion. 2011 11 4 705 731 10.1037/a0023980 21707162
Yik, M., Russell, J. A. & Steiger, J. H. A 12-point circumplex structure of core affect. Emotion. 11(4), 705–731 (2011).21707162 10.1037/a0023980
30. Ji C Mudiyanselage TB Gao Y Pan Y A review of infant cry analysis and classification EURASIP J. Audio Speech Music Process. 2021 2021 1 8 10.1186/s13636-021-00197-5
Ji, C., Mudiyanselage, T. B., Gao, Y. & Pan, Y. A review of infant cry analysis and classification. EURASIP J. Audio Speech Music Process. 2021(1), 8 (2021).10.1186/s13636-021-00197-5
31. Mottola CA Measurement strategies: The visual analogue scale Adv. Skin Wound Care 1993 6 5 56 66
Mottola, C. A. Measurement strategies: The visual analogue scale. Adv. Skin Wound Care 6(5), 56–66 (1993).
32. Bijur PE Silver W Gallagher EJ Reliability of the visual analog scale for measurement of acute pain Acad. Emerg. Med. 2001 8 12 1153 1157 10.1111/j.1553-2712.2001.tb01132.x 11733293
Bijur, P. E., Silver, W. & Gallagher, E. J. Reliability of the visual analog scale for measurement of acute pain. Acad. Emerg. Med. 8(12), 1153–1157 (2001).11733293 10.1111/j.1553-2712.2001.tb01132.x
33. Crellin DJ Harrison D Santamaria N Huque H Babl FE The psychometric properties of the visual analogue scale applied by an observer to assess procedural pain in infants and young children: An observational study J. Pediatr. Nurs. 2021 59 89 95 10.1016/j.pedn.2021.01.018 33561663
Crellin, D. J., Harrison, D., Santamaria, N., Huque, H. & Babl, F. E. The psychometric properties of the visual analogue scale applied by an observer to assess procedural pain in infants and young children: An observational study. J. Pediatr. Nurs. 59, 89–95 (2021).33561663 10.1016/j.pedn.2021.01.018
