
==== Front
PNAS Nexus
PNAS Nexus
pnasnexus
PNAS Nexus
2752-6542
Oxford University Press US

10.1093/pnasnexus/pgae370
pgae370
Physical Sciences and Engineering
AcademicSubjects/MED00010
AcademicSubjects/SCI00010
AcademicSubjects/SOC00010
PNAS_Nexus/bioeng
PNAS_Nexus/psych-bio
Eye tracking is more sensitive than skin conductance response in detecting mild environmental stimuli
https://orcid.org/0000-0002-9463-9782
Khazaei Saman Department of Biomedical Engineering, New York University, 433 1st Ave, New York, NY 10010, USA
Tech4Health Institute, NYU Langone Health, 433 1st Ave, New York, NY 10010, USA

https://orcid.org/0000-0001-5117-2628
Faghih Rose T Department of Biomedical Engineering, New York University, 433 1st Ave, New York, NY 10010, USA
Tech4Health Institute, NYU Langone Health, 433 1st Ave, New York, NY 10010, USA

Abbott Derek Editor
To whom correspondence should be addressed: Email: rfaghih@nyu.edu
Competing Interest: R.T.F. and S.K. are co-inventors of a patent application filed by the New York University based on this research.

9 2024
28 8 2024
28 8 2024
3 9 pgae37004 3 2024
08 8 2024
13 9 2024
© The Author(s) 2024. Published by Oxford University Press on behalf of National Academy of Sciences.
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (https://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact journals.permissions@oup.com.

Abstract

The skin conductance (SC) and eye tracking data are two potential arousal-related psychophysiological signals that can serve as the interoceptive unconditioned response to aversive stimuli (e.g. electric shocks). The current research investigates the sensitivity of these signals in detecting mild electric shock by decoding the hidden arousal and interoceptive awareness (IA) states. While well-established frameworks exist to decode the arousal state from the SC signal, there is a lack of a systematic approach that decodes the IA state from pupillometry and eye gaze measurements. We extract the physiological-based features from eye tracking data to recover the IA-related neural activity. Employing a Bayesian filtering framework, we decode the IA state in fear conditioning and extinction experiments where mild electric shock is used. We independently decode the underlying arousal state using binary and marked point process (MPP) observations derived from concurrently collected SC data. Eight of 11 subjects present a significantly (P-value <0.001) higher IA state in trials that were always accompanied by electric shock (CS+US+) compared to trials that were never accompanied by electric shock (CS−). According to the decoded SC-based arousal state, only five (binary observation) and four (MPP observation) subjects present a significantly higher arousal state in CS+US+ trials than CS− trials. In conclusion, the decoded hidden brain state from eye tracking data better agrees with the presented mild stimuli. Tracking IA state from eye tracking data can lead to the development of contactless monitors for neuropsychiatric and neurodegenerative disorders.

interoceptive awareness
eye tracking
pavlovian conditioning
skin conductance response
arousal
National Institutes of Health (NIH) 10.13039/100000002 R35GM151353 National Science Foundation (NSF) 10.13039/100000001 2226123
==== Body
pmcSignificance Statement

Decoding the underlying brain states in response to mild unpleasant stimuli (mild electric shock) can illuminate closed-loop system designs. In this research, we develop a framework to decode the state trajectory from the eye tracking data. In parallel, we employ a previously developed technique that decodes the state trajectory using the skin conductance (SC) signal. Our findings present a higher sensitivity of eye tracking data in detecting mild electric shock than the SC signal.

Introduction

Environmental stimuli consistently impact the internal brain states (1). The effects can be captured by decoding these hidden brain states of interest from different physiological signals. Particularly, the physiological tissues can deliver a signal to the central nervous system and convey information on the current state of the body (2). The nervous system sensation, interpretation, and integration of the internal body signals is often called interoception. Due to the complex anatomy of the interoceptive afferents, measuring the pure interoceptive awareness (IA) state in a noninvasive and nonsubjective manner is difficult (3, 4). Anatomically and functionally, there is an interplay between how the body processes the internal signals (pure interoception) and how it interprets external sensory information (5). To facilitate the modeling step, considering interoception and exteroception in the same framework is one of the potential approaches in contemporary computational neuroscience (5). In this research, we consider a broader sense of interoception that reflects the changes in brain activity in response to external stressors in the fear learning paradigm where a mild electric shock serves as an aversive stimulus. We investigate the SC and eye tracking data as two arousal-related psychophysiological signals that can serve as the interoceptive unconditioned response (UR) to aversive stimuli (e.g. electric shocks) (6, 7). We evaluate the sensitivity of these signals through the hidden arousal and IA states. In particular, we develop a decoder to track the IA state using pupillometry and eye gaze measurements. In parallel, we apply the previously developed frameworks to decode the arousal state from SC signal (8, 9).

Recent research underlines an association between interoception and eye (10, 11), while the IA-related eye tracking features need to be characterized. This motivates us to extract the IA-related features from the eye tracking data and quantify one’s IA in response to an environmental stimulus (electric shock). Findings on interoception and autonomic nervous system (ANS) activation suggest that in addition to arousal, interoception also evokes ANS activation (12). From the neurophysiology perspective, there is enough evidence to detect the influence of the ANS on the functions of the eye (13). Thus, the eye movements and pupil measures may serve as a candidate for decoding the mediating internal brain state. In particular, it has been noted that pupil measures are directly impacted by the ANS (14). The dilation and constriction of the pupil can be adjusted by the variations in light levels, cognitive factors, and ANS response (15). Mainly, two muscles are involved in changing the pupil diameter: the dilator pupillae and the sphincter pupillae. The dilator pupillae is controlled by the sympathetic nervous system (a branch of ANS), and the sphincter pupillae is governed by the parasympathetic nervous system (a branch of ANS).

As opposed to the pupil size, the eye position and orientation are governed mainly by the somatic nervous system (SNS). Particularly, the extraocular muscles are directly controlled by the SNS rather than ANS (16). Although eye movements are determined mainly by the SNS, the ANS still contributes enough to the eye position by affecting the blood flow and oxygen supply of extraocular muscles (13), making eye movements an informative feature in studying the internal brain states. Notably, the correlation between the fixation duration and attentional focus has been illustrated (15). The impact of autonomic activities on saccadic eye movements—rapid eye movements that suddenly change the fixation point (17)—may appear on the level of the excitatory burst neurons such that the neurons’ firing rates encode the saccades’ velocity signal (18).

In this research, given the presence of mild environmental stimuli, we are exclusively interested in recovering the IA-related neural activity in response to electric shock from the eye tracking data and decoding the IA state in the Pavlovian fear conditioning and fear extinction experiments. In the Pavlovian conditioning paradigm, subjects learn to predict aversive stimuli throughout the experiment (19). The aversive stimuli serve as the unconditioned stimuli (US). The unconditioned stimuli (US) can be paired with different types of neutral cues to serve as the conditioned stimuli (CS) (20). In this research, two types of visual CS have been used: CS+, which will be accompanied by US 50% of the time, and CS−, which will not be accompanied by US at all (US−). We consider a combination of the eye tracking features as an informative index of IA in order to decode the IA state.

The arousal state is another informative brain state in the fear conditioning and extinction paradigm that can reflect the underlying brain activity in response to the stimuli (21). The SC signal is a measure of electrodermal activity (EDA) widely used to decode the hidden arousal state (8). In the fear learning context, the SC signal can be seen as an interoceptive unconditioned response (UR) (6), and arousal may present a form of interoception (22); there is a strong positive association between IA and arousal (23). According to the wide spectrum of the presented interoception and the high positive correlation between arousal and interoception, we may expect a parallelized evoked response in arousal and interoception in response to the electric shock. The arousal state can be used to perform a qualitative analysis with respect to the decoded IA state.

The underlying arousal-related ANS activation can be recovered from the “spikey” nature of the SC signal (8). The onset and amplitude of the recovered ANS activation can be utilized to form various types of observation vectors and decode the hidden arousal state accordingly. In particular, the onset of recovered ANS activation can be employed to form a binary-type observation vector. In this research, we use the term “event” to describe the binary-type observation vectors. Thus, the term arousal event refers to the onset of recovered ANS activation. Inspired by the marked point process (MPP) framework, the amplitude of recovered ANS activation can be coupled with the point process arousal events to form the MPP-type observation vector (8). The MPP framework has been employed in neuroscience to relate the ensemble neural spiking activity to any relevant covariates (24). We employ binary-type and MPP-type observations derived from SC (EDA-based), and we use arousal decoders to estimate the EDA-based arousal state in fear conditioning and extinction experiments (8, 9). We compare the decoded IA state derived from eye tracking with estimated arousal from the SC signal.

Most of the illustrated eye tracking features have been employed in heterogeneous paradigms (25), and studies solely focus on pupillometry or eye movements. Also, the majority of proposed brain state decoders apply machine learning (ML) and deep learning (DL) algorithms, which require a training session beforehand. We develop a systematic approach that simultaneously considers pupillometry and eye gaze measurements to quantify the IA state. Particularly, we employ pupillometry and eye gaze to recover the brain’s IA-related neural activities (i.e. IA-related events). Then, using the Bayesian filtering approach within an expectation–maximization (EM) algorithm, we decode the eye-based IA state during fear conditioning and extinction experiments with no prior training session. We hypothesize that the derived IA state based on the eye tracking data maintains a higher level in the presence of environmental stimuli (i.e. mild electric shock) than the no-shock condition. We evaluate the eye-based findings by considering the SC signal as a metric of arousal and decode the arousal state (8). We report our findings and conclude that eye tracking data presents a more sensitive response to mild electric shock than the SC response. The findings of this study may contribute to the design of personalized closed-loop architectures for neuropsychiatric, neurological, and neurodegenerative disorders such as posttraumatic stress disorder (PTSD), stroke, and Alzheimer’s disease (AD) (6, 26).

Results

The recovered ANS activity

The SC signal is used to estimate the arousal state and compare the findings from the eye tracking measurements. Particularly, by modeling a sweat secretion process, the fast-varying component of the SC signal can be seen as a result of the ANS neural stimuli. By performing a signal deconvolution, the ANS activation can be recovered from the SC signal (27). Then, the ANS activation can be used to decode the underlying arousal state. The left column of Fig. 1 presents (A) the recorded SC signal and (B) the recovered ANS activation using a deconvolution framework for one subject during the experiment (27). A random window of the experiment in the right column of Fig. 1 depicts (A) the SC signal and (B) the recovered ANS activity. The background colors correspond to three types of trials: (1) CS+US+ trials (red) in which the electric shocks and the conditioned stimuli are both presented; (2) CS+US− trials (yellow) in which the electric shocks are absent while the conditioned stimuli are presented; (3) CS− trials (blue). The SC signal and recovered ANS activity for all participants are presented in the Supplementary material.

Fig. 1. An example of the recovered ANS activation from the SC signal (a measure of EDA) for one subject. The left column represents the complete recorded data, and the right column shows a random window of the experiment. The sub-panels, from top to bottom, present: A) the raw SC signal; B) the recovered ANS activation.

The recovered IA-related neural activity

The IA feature extraction step and the recovery of the IA-related neural activity from the eye tracking data are visualized in Fig. 2. The subplots located on the left column of Fig. 2 refer to the recorded data within the whole experiment, while the sub-panels on the right side present a random window of the experiment. In order to form a binary vector of the IA-related neural activity derived from the eye tracking measurements, the following features are considered: (A) the fixation onset and its associated duration, (B) the pupil size, (C) the pupil dilation speed, (D) the eye movement velocity, and (E) acceleration. The intersection of the time stamps that meet the specified conditions (covered in the methods section) would lead us to form the IA-related neural activity (binary) presented in sub-panel (F). The background colors refer to CS+US+ (red), CS+US− (yellow), and CS− (blue) trials. The determined conditions for each feature are set based on the experimental setup, and they can be defined as hyperparameters, which can vary. The feature extraction and observation formation steps for all subjects are depicted in the Supplementary material.

Fig. 2. An example of the eye-based feature extraction steps and the formation of the IA-related observation for one subject. The left column represents the complete recorded data, and the right column shows a random window of the experiment. The sub-panels, from top to bottom, present: A) the onset of the fixation (blue impulses) and the fixation duration (the amplitude of each impulse); B) the average pupil size (derived from both eyes); C) the pupil size derivative (dilation and constriction speed); D) the average velocity (derived from both eyes); E) the average acceleration (derived from both eyes); F) the recovered IA-related events from the eye (blue impulses).

The IA and arousal states’ responses to the presented stimuli

In Fig. 3, we analyze the epoch of signals over the trials with respect to the presented stimuli. Here, the epoch of a signal refers to the average data within the specific time windows extracted from the continuous signal. Since we are interested in the time window of trials, we study the average of signal segments across the trials of interest. This type of evaluation is inspired by the event-related potential (ERP) studies, which evaluate a brainwave or electrical activity in response to the stimuli (28). Here, three types of trials are presented: CS+US+, CS+US−, and US−. The event-related potential-like (ERP-like) study aims to characterize the signal’s response to the presented stimuli (29). The investigated signals in this study are presented in Fig. 3: (A) the epoch of the velocity (average of two eyes) for one example subject, (B) the epoch of the pupil size signal for one example subject, (C) the epoch of the SC signal for one example subject, (D) the epoch of the estimated IA state decoded from the IA-related neural activity (binary observation) for one example subject, (E) the epoch of arousal state decoded using the MPP observation extracted from the SC for one example subject, (F) the epoch of arousal state decoded from binary observation for one example subject, (G) the distribution of IA epochs among the subjects (11 subjects), (H) the distribution of MPP-based arousal epochs among the subjects (11 subjects), and (I) the distribution of binary-based arousal epochs among the subjects (11 subjects). The red colors correspond to the CS+US+ trials, the yellow ones correspond to the CS+US− trials, and the blue ones correspond to the CS- trials. The epoch of the presented signals for each individual is in the Supplementary material.

Fig. 3. The event-related potential-like (ERP-like) analysis in the fear conditioning and extinction experiments. The sub-panels of the figure present: A) the epoch of the average velocity and its 95% confidence limits across CS+US+ trials (red), CS+US− trials (yellow), and CS− trials (blue) for an example subject; B) the epoch of the pupil size and its 95% confidence limits across CS+US+ trials (red), CS+US− trials (yellow), and CS− trials (blue) for an example subject; C) the epoch of SC signal, and its 95% confidence limits across CS+US+ trials (red), CS+US− trials (yellow), and CS− trials (blue) for an example subject; D) the epoch of the decoded IA state from IA-related neural activities derived from the eye tracking data (eye-based), and its 95% confidence limits across CS+US+ trials (red), CS+US− trials (yellow), and CS− trials (blue) for an example subject; E) the epoch of the decoded arousal state from ANS activations and their amplitudes (MPP) derived from the EDA measurements, and the 95% confidence limits across CS+US+ trials (red), CS+US− trials (yellow), and CS− trials (blue) for an example subject. F) the epoch of the decoded arousal state from arousal events (binary), and its 95% confidence limits across CS+US+ trials (red), CS+US− trials (yellow), and CS− trials (blue) for an example subject. G) the box plots for the epoch of the decoded IA state from the eye tracking data (eye-based) across CS+US+ trials (red box), CS+US− trials (yellow box), and CS− trials (blue box) for all subjects; H) the box plots for the epoch of the decoded arousal state (MPP-based) from the EDA measurements across CS+US+ trials (red box), CS+US− trials (yellow box), and CS− trials (blue box) for all subjects; I) the box plots for the epoch of the decoded arousal state (binary-based) from the EDA measurements across CS+US+ trials (red box), CS+US− trials (yellow box), and CS− trials (blue box) for all subjects. The *** is used to indicate P<0.001 where the findings are statistically significant.

Given the decoded IA and arousal states and considering the epochs of CS+US+ trials (N=30 trials), CS+US− trials (N=50 trials), and CS− trials (N=80 trials), we perform the one-tailed Wilcoxon signed-rank test for each individual as well as participant pool (N=11), and the significance levels are set as P≤0.001 (see Statistical Analysis section). For 8 out of 11 subjects, the epoch of IA state presents a significantly higher level of IA state within CS+US+ trials compared to CS− ones (Table 1). For the arousal state case, using the binary and MPP-type decoders, five and four subjects display a significantly higher level of state within CS+US+ trials compared to CS− ones, respectively (Table 1). Considering the aggregated state values among the subjects, the epoch of IA state within CS+US+ trials is significantly higher than CS+US− and CS− trials (Fig. 3G).

Table 1. The one-tailed Wilcoxon signed-rank test for the decoded IA and arousal states across the CS+US+ and CS− trials.

subject	Eye-based IA state	EDA-based arousal sate (binary)	EDA-based arousal state (MPP)	
	CS+US+ (mean)	CS− (mean)	CS+US+>CS−  P-value	CS+US+ (mean)	CS− (mean)	CS+US+>CS−  P-value	CS+US+ (mean)	CS− (mean)	CS+US+>CS−  P-value	
1	− 0.0688	− 0.0020	–	− 0.0284	− 0.0037	–	− 0.1945	0.0544	–	
2	− 0.1052	0.0402	–	− 0.0641	− 0.0714	P<0.001***	− 0.0499	− 0.1232	P<0.001***	
3	− 0.0217	− 0.1963	P<0.001***	0.0350	− 0.0016	P<0.001***	− 0.0414	0.0284	–	
4	0.0869	− 0.0858	P<0.001***	− 0.0662	− 0.0162	–	− 0.2574	0.0813	–	
5	− 1.2971	− 0.6591	–	− 0.2374	− 0.1539	–	− 0.3809	− 0.1811	–	
6	0.1120	− 0.0729	P<0.001***	− 0.0337	− 0.0064	–	− 0.1337	0.0705	–	
7	0.1606	− 0.3505	P<0.001***	− 0.0182	− 0.0033	–	− 0.0617	0.0115	–	
8	0.1844	− 0.2618	P<0.001***	− 0.5254	− 0.4766	–	− 0.7411	− 0.8041	P<0.001***	
9	0.5674	− 0.6531	P<0.001***	0.1416	− 0.0481	P<0.001***	0.3191	− 0.1129	P<0.001***	
10	− 0.0793	− 0.2049	P<0.001***	0.0087	− 0.0037	P<0.001***	− 0.0512	0.0136	–	
11	0.2124	− 0.1542	P<0.001***	0.1319	− 0.0059	P<0.001***	0.0617	− 0.0163	P<0.001***	
This table demonstrates the average epochs of the decoded eye-based IA state from the binary observation and the decoded and EDA-based arousal state from the binary and MPP observations. The one-tailed Wilcoxon signed-rank test is performed. The *** is used to indicate P<0.001 where the findings are statistically significant.

In Table 1, we report the P-values for each individual given the null hypothesis, h0: the epochs of arousal state (derived from binary observation and MPP observation) and IA state within CS+US+ trials maintain values that are either equal or less than the epochs of arousal and IA states within CS− trials (h0:CS+US+≤CS−). Similar statistical analysis for other types of trials can be found in the Supplementary material, where we perform a one-tailed Wilcoxon signed-rank test to evaluate the h0:CS+US+≤CS+US− and h0:CS+US−≤CS−. We report the corresponding P-values in the Supplementary material.

The decoded IA and arousal states

The hidden brain state of interest can be decoded from the recovered observation vector. To decode the hidden state independent of prior training sessions, the Bayesian state-space framework within the EM algorithm can be utilized (8). We present each decoder based on the input observation, which can be binary-type or MPP-type data formed from the EDA measurements (EDA-based), and binary-type formed from the eye tracking measurements (eye-based). The sub-figures in Fig. 4, from left to right, demonstrate the estimates of IA, the estimates of arousal according to the extracted MPP observation, and the estimates of arousal given the binary-type observation, respectively. The IA decoder utilizes the recovered IA-related neural activity to estimate the IA matrices, while the arousal decoder takes the recovered ANS activity from EDA to decode the arousal. Subplot B of each sub-figure denotes the type of applied observation; subplot C of each sub-figure indicates a decoded state, and subplot D of each sub-figure presents the probability of observing the impulses. The quantified brain states vary between the subjects. Hence, there should be a general tool to personalize the state and express the index of arousal and IA. The concept of high arousal index (HAI) has been introduced in (8), where the HAI can be derived from a person’s baseline, HAI=Pr(zj>median(z1:J)). Thus, HAI is a person-specific measurement between 0 and 1 to generalize the arousal level. Inspired by HAI derivation, we generalize the interoception level of the subject during the whole experiment by deriving the high interoception index (HII) (8). The HAI and HII are shown in subplot E. The black background colors indicate the CS+US+ trials, the yellow ones refer to the CS+US– trials, and the blue ones corresponds to the CS– trials. The estimated IA and arousal states for all subjects are presented in the Supplementary material.

Fig. 4. An example of the decoded IA and arousal states for one subject. The left column represents the estimates of the eye-based IA using a binary observation (IA-related neural activity), the middle column corresponds to the estimates of the EDA-based arousal derived from MPP observation, and the right column depicts the EDA-based estimates of arousal from a binary observation. The sub-panels, from top to bottom, depict: A) the physiological signal of interest (e.g. pupil size or SC); B) the applied observation vector; C) the decoded state trajectory its 95% confidence limits; D) the probability of event occurrence; E) the internal brain state index (HII or HAI).

Discussion

Although several neural pathways contribute to experiencing interoceptive sensing in response to stimuli, eye movement is one of the physiological signals that appear to be directly involved in reflecting the interoceptive UR to aversive stimuli (e.g. electric shocks) (6). Here, a comprehensive pipeline is provided that employs pupillometry and eye gaze to decode the underlying IA in the fear conditioning and fear extinction experiments. The absence of ground truth in the context of brain state decoding leads us to an implicit evaluation. Given the high association between the bodily arousal states and IA state (22, 30), we decode the arousal state from the skin conductance signal (a measure of EDA) and compare the eye-based IA with EDA-based arousal. The studied experiment has three types of trials. We decode and display the IA and arousal states within each type of trial.

IA-related feature selection

Figure 2 depicts the extracted features and recovered IA-related events from the eye tracking data. To recover the IA-related neural activity, we extract the informative eye tracking features and form the sets of bins. The first considered feature is the eye fixation and its duration (subplot A). The fixation duration varies from several milliseconds to several seconds (31). Here, we are interested in the long fixation duration. We apply certain thresholds to sort out the bins based on the empirical evidence, the person’s baseline, and the studied experiment. We set the fixation duration threshold to 0.35 seconds. The pupil size is the next informative feature to be employed. The normal pupil size in adults varies from 2 mm to 8 mm (1). In subplot B of Fig. 2, the pupil size takes values in the range of 2–6 mm for this particular subject. We consider 3 mm as the threshold to classify the large pupil sizes. This large pupil size is in accordance with expected pupil dilation in response to arousing conditions and receiving shock (32). We further investigate the pupil dilation speed by finding the derivative of pupil size throughout the experiment and forming bins where the derivative of pupil size passes the 0.2 mm/s threshold (subplot C). According to subplot D of Fig. 2, the sharp increase in velocity can be detected in response to the stimuli, while during the inter-trial interval (ITI), the velocity decreases. To detect the sharp increases in velocity, we set 60 ∘ /s as the threshold. The acceleration amplitude is considered in the fixation detection step where the acceleration amplitude for a fixation event is aj<3,000∘/s2 (33, 34). We recover the IA-related neural activity according to the described features and thresholds (subplot F).

The SC as an index of arousal

The SC response is a well-established index of arousal that can be used to quantify the arousal state (21). Several data-driven and model-based techniques can be employed in analyzing the SC signal. As an instance of traditional data-driven approaches, we can refer to evaluating the peak values of SC as the informative indicator of arousal (35, 36). Another technique is to use the SC level within a specific time window as an arousal indicator (37). While these methods can be informative regarding the underlying arousal, they are limited to one particular aspect of the data, and some signal components are left unused (38). For instance, the SC can be decomposed into slow-varying (i.e. tonic) and fast-varying components (i.e. phasic), where the phasic component may be applied as one of the informative observations regarding the mental state of a person (27). Also, it should be noted that these approaches are not analytical (38), and they are highly dependent on the studied data.

On the other hand, to address the raised concerns and recover the arousal-related information from the raw SC signal, several model-based and analytical algorithms have been developed (27, 39–43). Most of these model-based approaches consider a causal model and perform the sparse deconvolution to recover the arousal-related brain input. Here, we consider the previously developed physiologically motivated model in (27) to recover the phasic signal and arousal-related brain input from the data. Then, we track the continuous arousal state trajectory from the recovered arousal events via the Bayesian state-space approach (8, 9). The employed deconvolution and state estimation approach in this research has been tested in various experimental paradigms in the presence of cognitive tasks, auditory stimuli, and fear conditioning experiment (8, 27). However, it is essential to highlight that the employed approaches may impose a high degree of freedom, which may result in overfitting. Also, the algorithms used for EDA data outperform in the presence of minimal artifacts. To further test the reliability and robustness of the employed EDA analysis algorithms, studies need to consider the presence of subjective stimuli as well as annotations (44), and validate the employed arousal event recovery as well as estimation approach.

The physiological signal alignment with the presented electric shock as the unconditional stimuli

The SC signal has been widely studied in Pavlovian conditioning experiments to evaluate the autonomic response to the stimuli (21, 45). In (21), a Bayesian state-space model and decoder were developed, and the hidden arousal state was decoded from the available SC and heart rate signal within the trace conditioning experiment. It has been shown that the SC signal and the decoded arousal were elevated within the CS+US+ trials where the electric shock was presented. Inspired by (8), we analyze our findings with respect to each type of trial. Notably, in Fig. 3, the ERP-like perspective for one participant reveals high levels of IA state in the presence of mild electric shock. In contrast, the decoded arousal states from both MPP and binary observations maintain lower values within CS+US+ trials. The observed trend within the epochs of the arousal state, derived based on both binary and MPP observations, agrees with the epochs of the SC signal. The observed high IA estimates during the CS+US+ trials conform with the epochs of pupil size associated with CS+US+ trials after the electric shock onset.

In the previous study in (21), the SC signal presented a reliable index of autonomic response to an intense electric shock, while the raw eye tracking data was not available to be investigated. In this study, we investigate the SC signal and eye tracking measurements in the presence of milder electric shock. We observe that the eye tracking data reveals more sensitivity to the mild electric shock than SC. It should be noted that the previously studied experiment in (21) was a trace fear conditioning experiment in the presence of more intense electric shock (90% of the participant’s threshold), whereas the studied experiments in this research include fear conditioning and fear extinction paradigm in the presence of milder shock. Perhaps SC data is not very sensitive to mild stimuli (46). Also, trace conditioning engages the hippocampus, and delay conditioning mainly engages the amygdala (21).

Our findings present a more sensitive physiological response of the eye compared to the sudomotor activity. While multiple factors can contribute to the observed trend, one may consider the neural pathways and anatomical structures involved in the physiological response as the influential factors. In particular, the sweat gland activity includes a long path with a multistep process (27), which may result in a relatively more extended latency response compared to the eye movements (47, 48), which are directly controlled by the oculomotor system with the specialized circuitry that offers a precise and rapid response to the stimuli (49). It should be noted that physiological systems that are highly sensitive to the stimuli tend to present a shorter response latency (50). Eye tracking can be an informative and sensitive measure of brain activity in sensorimotor and cognitive processes. This agrees with the finding in (37), where the pupil size presented a higher sensitivity to the induced mental workload compared to the SC data.

One of the essential advantages that both modalities offer is that they both can be collected in a noninvasive manner and in everyday life settings, which makes both signals suitable for seamless brain monitoring. Another advantage that these signals share is that both signals are recognized interoceptive signals with rich information (3, 7). Specifically, pupilation can reflect interoceptive UR activities in the context of fear conditioning (6).

While both signals are informative and can be collected simply, their implementation does come with specific challenges. One of these challenges is the lack of preciseness and robustness compared to direct brain recording. However, researchers can address this by combining these modalities with direct brain measurements such as electroencephalogram (EEG), evoked potentials, and Functional magnetic resonance imaging (fMRI), thereby improving the preciseness and robustness for implementation within closed-loop architectures. Another challenge is the presence of artifacts in these signals, which may be addressed with the recent advancements in biomedical sensor design procedures.

It is important to highlight that the applied visual stimulation may impact the brain state response detected through the studied physiological signals. Particularly, the employed eye tracking features might be more prone to being affected by visual stimuli. Hence, further studies with various types of stimuli such as auditory stimulation (20, 51) can reduce the potential confounding factors and provide a better insight into the physiological response to the electric shock.

Also, we investigated a relatively low sample size, which may limit the generalizability of the findings. We may focus on the studied individual and interpret the findings. Further studies with larger sample sizes and in the presence of various behavioral experiments can provide a comprehensive overview. Specifically, we mainly consider the fear conditioning experiment and investigate the electric shock as the stimuli. The electric shock may not solely span the entire spectrum of the studied brain states through the investigated physiological signals. Hence, we should be cautious when interpreting the findings. To gain a better understanding of how physiological signals such as SC and eye tracking data can reflect the brain’s response to external stimuli, further investigations in the presence of tactile, auditory, and olfactory stimulation can be performed (51).

The Underlying Brain State Estimation from the Available Physiological Observation:

Figure 4 indicates the eye-based IA state and the EDA-based arousal state estimations for one subject. By comparing the EDA-based arousal state estimation results, we notice that the arousal state derived from the MPP-type observation better agrees with the electric shock onset. Specifically, the sharp increases in the decoded state via MPP observation are noticeable within most of the CS+US+ trials, while the decoded arousal from the arousal events (binary) does not reveal a similar alignment. This underlines the importance of accounting for the ANS activation amplitude, and would motivate us to consider the relevant covariates of IA-related events in forming the observation for future IA state decoders. Comparing the arousal and IA estimates derived from binary observation, the decoded arousal state has a smoother trajectory while binary observation is fed to both decoders. The smoothness of the estimated state can be related to the sparsity level of the observation vector, the noise dynamic of the state model, the brain state of interest, and the applied physiological signal. Hence, selecting a physiological signal, extracting features, and processing steps can play a crucial role in this paradigm.

In this research, we propose a methodology to monitor a hidden IA state from the eye tracking data. The core of our approach is the recovery of the underlying brain input from the eye tracking data, which allows us to decode the trajectory of the hidden brain state. In parallel, we use skin conductance, a well-established measure of arousal, to decode the arousal state and evaluate the interoceptive UR to aversive stimuli. We study a total of 11 participants and perform an in-depth analysis considering 30 CS+US+ and 50 CS+US− trials for each person. Our findings, viewed from an individualized perspective, suggest that eye tracking is more sensitive than SC response in detecting mild electric shock. We must note that we study a relatively small sample size. This may limit the generalizability of the findings, and we should avoid any overinterpretation. Instead, it should be noted that findings are limited to the studied individuals. To have a decisive resolution about the sensitivity of the employed signal, future investigations with large sample sizes, the presence of direct brain measurements, various types of stimulation, and the inclusion of a control group are necessary.

Additionally, we mainly investigate electric shock as the stimuli, and it may not span the entire spectrum of the studied brain states, which are identifiable through eye tracking and skin conductance. Hence, we should be cautious when generalizing the findings. To gain a better understanding of how physiological signals such as SC and eye tracking data can reflect the interoceptive and arousal responses, further investigations in the presence of various internal and external stimulation (e.g. visual, auditory, tactile, gustatory, and olfactory) are required.

While the designed IA decoder has the capacity to provide groundbreaking insights into the interoception decoding procedures, it is worth highlighting that the developed methodology can be further improved. Particularly, future works may employ advanced preprocessing and smoothing frameworks such as Hampel and Savitzky-Golay filters (52, 53). Previous studies in (35, 54) found the larger startle blink within the aversive stimuli and identified the blink reflex as an indicator of arousal. Also, it has been observed that the startle blink reflexes are larger when processing cues paired with aversive shock (55). Hence, the blink reflex may potentially serve as an arousal-related signal to be investigated and may be an informative index of brain response in this paradigm (56). Thus, this study does not exclude the blink and fast and slow dilation speed samples from the data. However, at this point, we do not directly incorporate the blink reflex in forming the observation vector, and the blink reflex can be evaluated in future studies where facial electromyography sensors are employed.

Also, when forming the IA-related observation, we set the thresholds manually and avoid the ML-based approaches, which require training sessions; however, we could implement a training step and find thresholds based on the trained model (57). Here, the applied thresholds preserve the generalizability among the subjects. Nevertheless, with some caveats, they can become personalized by considering person-specific conditions, which requires further investigations on the pupil size baseline. Furthermore, similar to the SC signal, the pupil size signal can be observed as a summation of multiple components, namely tonic (slow-varying) and phasic (fast-varying) components (58), and a physiologically based framework can be developed to separate them and recover the brain input from the phasic component. Another point to consider in the future is to create a multi-input, single-output decoder that utilizes both eyes in a concurrent fashion and performs the decoding accordingly.

The proposed framework, tested on previously collected data from healthy subjects, is ready for the next crucial step. To truly expand the impact of this research on neuropsychiatric and neurodegenerative disorders, we must test the algorithm on patients in the presence of different interventions. This will validate the decoder’s practicality and open up new possibilities for its application. Specifically, the decoder should be implemented within the personalized closed-loop architecture, and the outcome should be evaluated with meticulous care.

Limitations of the study

There are limitations to our study, and we should be cautious in generalizing the findings. As the Bayesian EDA deconvolution framework developed in (27) presents an enhanced performance with respect to other existing deconvolution methods, we did not use other EDA deconvolution frameworks such as (39–43). Also, we did not utilize conventionally used EDA feature extraction-based methods. Instead, we applied a system-theoretic deconvolution approach. In particular, we follow the EDA deconvolution framework utilized and validated in (27, 59–66) for EDA analysis in the presence of environmental stimuli (e.g. auditory stimulation). Then, we follow the Bayesian inference framework developed and validated in various experimental settings (8, 21, 36, 67–71). However, none of these previous studies included concurrently collected eye tracking data to be able to compare EDA and eye tracking. Future studies need to compare the findings based on both system-theoretic deconvolution approach and feature-extraction-based EDA analysis approaches.

Our findings, while valuable, are not without the limitations. Particularly, a relatively small sample size is considered, in the presence of potential confounding factors, and without a control group. These points can restrict the broader relevance of the findings. Hence, future research needs to be performed with larger sample size and presence of control group.

Another limitation to be noted is that we only focus on the external electric shock as the stimuli, and this may restrict the spectrum of state responses to the stimuli detectable through employed physiological signals. To address this limitation, other forms of internal and external stimulation should be considered in the future (72).

Our focus on SC and eye tracking data provides valuable insights into the interoceptive UR. However, to truly understand the complex phenomenon of interoception, we need to broaden our perspective by employing multidimensional measurements (73). Incorporating direct brain measurements such as EEG, evoked potentials, and fMRI alongside our current data will give us a more comprehensive view.

Also, it is worth highlighting that this research is at an early development stage, and there is a need for further considerations and innovations in this paradigm. As highlighted earlier, the proposed feature extraction step can be more systematic and personalized. Also, in this investigation phase, we are using data from the average of both eyes, which can increase the likelihood of information loss. One of the limitations that needs to be addressed in future studies is the absence of annotated states, which may be interpreted as an informative indicator of ground truth and can assist the evaluation of decoder output. Lastly, we only focus on the decoding part without the incorporation of a control strategy. Executing a proper control strategy should be listed as one of the future avenues of our research, which can broaden the impact and scope.

Conclusion

In conclusion, the eye seems to be an informative metric of the hidden brain state in response to mild environmental stimuli. The IA state derived from the eye tracking data is better aligned with the presented stimuli than the arousal state derived from the SC signal. Findings suggest that in the context of fear learning, the pupillometry and eye gaze measurements may be employed together to extract the applicable features, recover the IA-related neural activity, and decode the hidden brain state within everyday life settings.

Materials and methods

Participants and experimental procedure

We employ physiological signals collected within the fear conditioning and fear extinction experiments. The governmental research ethics committee in Switzerland approved the experimental procedures, and all the participants provided written informed consent. A detailed description of the studied dataset is available in (20, 74). The recorded signals include the SC response, electrocardiogram (ECG), respiration, pupil size, and gaze coordinates measurements collected from 29 healthy participants (17 females and 12 males) performing a classical (Pavlovian) discriminant delay fear conditioning and extinction experiments (20). 18 participants were excluded from this study due to a lack of UR, absence of fear conditioning occurrence, measurement error, and artifacts. Hence, a total number of 11 subjects (subjects 1 to 11) are investigated. A list of participants’ information with the exclusion criteria is available in the Supplementary material. In this experiment, two types of visual conditioned stimuli (CS+ and CS−) were presented. The applied visual stimuli were patterns with different colors and roughly equal brightness, contrast, and spatial frequencies. A 500-ms train of 250 square electric pulses was used as the unconditioned stimuli (US - electric shock). The intensity of US used during experiments was set based on the 85% pain threshold of participants. The CS+ was followed by the US half of the time (CS+US+) while the other half of the time, the electric shock was absent (CS+US−). The CS− was not followed by the US at all. The experiment consisted of 3 major blocks: The first two blocks—acquisition phase—included 15 CS+US+ (CS+ with shock) trials, 15 CS+US− (CS+ without shock) trials, and 30 CS− trials within each block; the last block—extinction phase—included 20 CS− and 20 CS+ trials without US occurrence. The order of trials was randomized in each block. Within each trial, the CS was displayed within the first 4 seconds, followed by the mild electric shock (if the US exists). The inter-trial interval (ITI) was randomized to be between 7 and 11 seconds.

Data acquisition and preprocessing

The gaze direction coordinates and pupil sizes were recorded by EyeLink 1,000 at a sampling rate of 500 Hz. The SC responses were collected at a sampling rate of 1,000 Hz from the thenar/hypothenar of the nondominant hand using Biopac system electrodes (20).

Inspired by the proposed framework in (20, 75), to filter out invalid gaze points and pupillometry samples, we consider two categories: (1) temporally isolated samples and (2) trend-line deviation outliers. Additionally, we predefine a feasible range of pupil size and filter out the out-of-range data by specifying 9 mm as the upper bound of the pupil size and 1.5 mm as the lower bound. The process of invalid sample removal generates nonequidistant gaps between data. To fill the gaps and increase the smoothness of the data, we implement the interpolation as described in (20, 75). The average range of saccade duration has been noted as 20–200 ms (76, 77). Thus, we downsample the data with a sampling frequency as low as 60 Hz (fs=60) to ensure the detection of saccadic movements (76, 77). Also, we apply the zero-phase low-pass filter with a cutoff frequency of 4 Hz to denoise the pupil size (75).

The SC raw signal can be considered as a summation of a slow-varying component (i.e. tonic) and a fast-varying component (i.e. phasic) (27). By modeling the phasic component as the convolution between the neural impulse train and the physiological system response, the ANS activation can be recovered (27). To recover the underlying ANS activation, we first preprocess the data and employ the high-order FIR low-pass filter with a 1 Hz cutoff, and then we downsampled the data to 4 Hz (27); inspired by Amin and Faghih (27), we use the proposed physiology-motivated state-space model as well as the EM-based deconvolution approach, and we recover the ANS activation from the signal.

Eye-based feature extraction

Given the gaze data in the x and y directions (Xj,Yj), we find the velocity amplitude (vj) and acceleration amplitude (aj) of the eye movements (33, 34):

(1) vj=(Xj−Xj−1)2+(Yj−Yj−1)2tj−tj−1aj=vj−vj−1tj−tj−1,

where t refers to the time, and j stands for the index (j=tfs). For the conversion of pixels to degrees, based on the experimental setup, we use 0.024 as the conversion factor. Following the fixation detection technique in (33, 34), and based on the nature of these data, we consider three conditions to detect the fixation of the eye: (1) vj<240∘/s, (2) aj<3,000∘/s2, and (3) fixation duration >100ms. Meeting these three conditions at the same time would lead to fixation detection. We are primarily interested in the fixation onset and duration, and we consider the rest of the eye movements as saccadic movements while we could employ more advanced grouping by accounting for noise and movements such as smooth pursuit, microsaccades, and glissade movements (17, 34, 78).

Proceeding with our hypothesis, to form an IA-related binary observation nj, we examine the information from pupil size (s), pupil dilation/constriction speed (dsdt), fixation duration (τf), and saccadic velocity. The binary observation nj is supposed to be an indicator of IA-related neural activity within an individual’s circuits in response to arousing inputs. As mentioned before, pupil dilation, an increase in the velocity of eye movements, and long fixations can all contribute to expressing the integration of input stimuli. Thus, by setting the thresholds (α1,α2,α3,α4) and considering the window size of interest (Δt→δ), we form the observation vector n. The window of interest (δ) starts with a fixation movement (τf) followed by a saccadic movement (τs) where the following conditions are satisfied:

(2) Δt=δ=τf+τs

(3) τf>α1

(4) sΔt→δ>α2

(5) dsdt>α3

(6) Max(vΔt→δ)>α4.

Once all the above statements hold, the start of the particular window (i.e. fixation onset) is marked as the IA-related neural activity where nj=1; otherwise, nj=0. In this study, the thresholds are set based on the dataset of interest and subjects while they are adaptive hyperparameters: α1=0.35s,α2=3mm,α3=0.2mm/s, α4=60∘/s, and they can take different values.

IA state-space model

Random walk models have been used widely to represent hidden brain states (8). To do so, we model the IA state zj as

(7) zj=zj−1+ϵj,

where ϵj∼N(0,σϵ2) is a process noise, and σϵ2 needs to be determined.

Following the approach presented in (9), we assume the formed binary observation (nj) follows the Bernoulli distribution with a probability mass function of pjnj(1−pj)1−nj, where pj=P(nj=1). To link the pj to zj, a logit transformation can be applied (9).

(8) log(pj1−pj)=λ+zj⇒pj=11+e−(λ+zj)

where constant λ can be determined by setting zj≈0 and p0=Σj=1j=JnJ (the average probability of nj=1) (9):

(9) λ≈log(p01−p0).

IA decoder

Similar to (9, 79), to derive a decoder given the binary-type observation vector, we employ the EM framework. The EM framework consists of the E-step and M-step where the E-step mainly focuses on the filtering, and the M-step determines z0 and σϵ2 given the E-step output:

E-step:

The E-step includes a forward filter and a backward smoother. In the forward filtering part, we employ the Bayesian filtering approach to estimate the hidden state (zj) (9, 79).

Predict:(10) zj|j−1=zj−1|j−1

(11) σj|j−12=σj−1|j−12+σϵ2

Update:(12) zj|j=zj|j−1+σj|j−12[nj−11+e−(λ+zj|j)]

(13) σj|j2=[1σj|j−12+e(λ+zj|j)(1+e(λ+zj|j))2]−1

Once j=J (last data point), by reversing the direction, we implement the smoother where zj|J and σj|J2 denote the smoothed state and variance, respectively.(14) Aj=σj|j2σj+1|j2,

(15) zj|J=zj|j+Aj(zj+1|J−zj+1|j),

(16) σj|J2=σj|j2+Aj2(σj+1|J2−σj+1|j2).

M-step: In the M-step, we estimate the unknown terms (σϵ2,z0) such that they maximize the expected value of the following log-likelihood function (E[Q]) (9, 79):(17) E[Q]=−J2log(2πσϵ2)−∑j=1JE[(zj−zj−1)2]2σϵ2.

Hence, the unknown parameters can be derived from following equations (9, 79).(18) σϵ2=2J+1[∑j=2J(σj|J2+zj|J2)−∑j=2J(Ajσj|J+zj|Jzj−1|J)]+1J+1[32z1|J2+2σ1|J2−(σJ|J2+zJ|J2)]

(19) z0=12z1|J.

The algorithm iterates between the E-step and the M-step until it satisfies the convergence criteria.

In order to decode the arousal state xj, based on two different types of observations, we consider two developed decoders in (9) and (8): (1) We use the recovered occurrence of ANS activation (binary) as the available observation, and we decode the underlying arousal state (9); (2) We employ the MPP framework and form the observation based on the occurrence of ANS activation (point process) as well as the amplitude of ANS activation (marked); then, we decode the hidden arousal state accordingly (8). A detailed description of arousal state decoders is provided in the Supplementary material.

Statistical analysis

Given the decoded IA and arousal states and considering the epochs of CS+US+ trials (N=30 trials), CS+US− trials (N=50 trials), and CS− trials (N=80 trials), we perform the one-tailed Wilcoxon signed-rank test for each individual as well as participant pool (N=11), and the significance levels are set as P≤0.001. In Table 1, we report the P-values for each individual given the null hypothesis, h0: the epochs of arousal state (derived from binary observation and MPP observation) and IA state within CS+US+ trials maintain values that are either equal or less than the epochs of arousal and IA states within CS− trials (h0:CS+US+≤CS−). Similar statistical analysis for other types of trials can be found in the Supplementary material, where we perform a one-tailed Wilcoxon signed-rank test to evaluate the h0:CS+US+≤CS+US− and h0:CS+US−≤CS−. We report the corresponding P-values in the Supplementary material.

Supplementary Material

pgae370_Supplementary_Data

Acknowledgments

The authors thank the anonymous reviewers for their valuable suggestions.

Supplementary Material

Supplementary material is available at PNAS Nexus online.

Funding

This work was supported in part by National Institutes of Health (NIH) grant R35GM151353 - MESH: Multimodal Estimators for Sensing Health, in part by National Science Foundation (NSF) under Grant 2226123 - Faculty Early Career Development Program (CAREER): MINDWATCH: Multimodal Intelligent Noninvasive brain state Decoder for Wearable AdapTive Closed-loop arcHitectures, and in part by New York University (NYU) start-up funds to R.T.F.

Author Contributions

S.K. and R.T.F. designed the methodology and participated in writing and reviewing the manuscript. S.K. performed the data curation, formal analysis, visualization, and software for this study. R.T.F. conceptualized the study, performed the funding acquisition, and provided the supervision.

Data Availability

All data generated or analyzed during this study are presented in this main manuscript and its Supplementary Material files. The raw dataset used in this study is publicly is publicly available through the Zenodo database (74), and it can be accessed at: https://doi.org/10.5281/zenodo.4290767. The code will be publicly available on computational medicine lab GitHub after publication.
==== Refs
References

1 Walker  HK, Hall  WD, Hurst  JW. 1990. Clinical methods: the history, physical, and laboratory examinations. 3rd. Boston (MA): Butterworths.
2 Cameron  OG . 2001. Interoception: the inside story-a model for psychosomatic processes. Psychosom Med. 63 (5 ):697–710.11573016
3 Khalsa  SS , et al. 2018. Interoception and mental health: a roadmap. Biol Psychiatry Cogn Neurosci Neuroimaging. 3 (6 ):501–513. 10.1016/j.bpsc.2017.12.004 29884281
4 Quigley  KS, Kanoski  S, Grill  WM, Barrett  LF, Tsakiris  M. 2021. Functions of interoception: from energy regulation to experience of the self. Trends Neurosci. 44 (1 ):29–38. 10.1016/j.tins.2020.09.008 33378654
5 Engelen  T, Solcà  M, Tallon-Baudry  C. 2023. Interoceptive rhythms in the brain. Nat Neurosci. 26 (10 ):1670–1684. 10.1038/s41593-023-01425-1 37697110
6 Joshi  SA, Aupperle  RL, Khalsa  SS. 2023. Interoception in fear learning and posttraumatic stress disorder. Focus. 21 (3 ):266–277.37404967
7 Chen  WG , et al. 2021. The emerging science of interoception: sensing, integrating, interpreting, and regulating signals within the self. Trends Neurosci. 44 (1 ):3–16. 10.1016/j.tins.2020.10.007 33378655
8 Wickramasuriya  DS, Faghih  RT. 2020. A marked point process filtering approach for tracking sympathetic arousal from skin conductance. IEEE Access. 8 :68499–68513. 10.1109/ACCESS.2020.2984508
9 Wickramasuriya  DS, Amin  M, Faghih  RT. 2019. Skin conductance as a viable alternative for closing the deep brain stimulation loop in neuropsychiatric disorders. Front Neurosci. 13 :780. 10.3389/fnins.2019.00780 31447627
10 Imafuku  M, Fukushima  H, Nakamura  Y, Myowa  M, Koike  S. 2020. Interoception is associated with the impact of eye contact on spontaneous facial mimicry. Sci Rep. 10 (1 ):19866. 10.1038/s41598-020-76393-8 33199748
11 Allen  M, Levy  A, Parr  T, Friston  KJ. 2022. In the body’s eye: the computational anatomy of interoceptive inference. PLoS Comput Biol. 18 (9 ):e1010490. 10.1371/journal.pcbi.1010490 36099315
12 Ueno  D, Ohira  H, Narumoto  J. 2023. Interoception and the autonomic nervous system: investigating affect, decision-making, and mental health. 10.3389/fnins.2022.1130324
13 McDougal  DH, Gamlin  PD. 2015. Autonomic control of the eye. Compr Physiol. 5 (1 ):439. 10.1002/cphy.c140014 25589275
14 Skaramagkas  V , et al. 2021. Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Rev Biomed Eng. 16 :260–277. 10.1109/RBME.2021.3066072
15 Eckstein  MK, Guerra-Carrillo  B, Singley  ATM, Bunge  SA. 2017. Beyond eye gaze: what else can eyetracking reveal about cognition and cognitive development?  Dev Cogn Neurosci. 25 :69–91. 10.1016/j.dcn.2016.11.001 27908561
16 Ju  Dr. W, Santos  A, Freeman  A, Daniele  E. 2018. Neuroscience. Toronto (ON):  eCampus.
17 Purves  D , et al. 2001. Neuroscience. 2nd ed. Sunderland (MA): Sinauer associates. Types of Eye Movements and Their Functions.
18 Sparks  DL . 2002. The brainstem control of saccadic eye movements. Nat Rev Neurosci. 3 (12 ):952–964. 10.1038/nrn986 12461552
19 Maren  S . 2001. Neurobiology of pavlovian fear conditioning. Annu Rev Neurosci. 24 (1 ):897–931. 10.1146/annurev.neuro.24.1.897 11520922
20 Xia  Y, Melinscak  F, Bach  DR. 2021. Saccadic scanpath length: an index for human threat conditioning. Behav Res Methods. 53 (4 ):1426–1439. 10.3758/s13428-020-01490-5 33169287
21 Wickramasuriya  DS, Faghih  RT. 2020. A mixed filter algorithm for sympathetic arousal tracking from skin conductance and heart rate measurements in pavlovian fear conditioning. PLoS One. 15 (4 ):e0231659. 10.1371/journal.pone.0231659 32324756
22 Fujimoto  A, Murray  EA, Rudebeck  PH. 2021. Interaction between decision-making and interoceptive representations of bodily arousal in frontal cortex. Proc Natl Acad Sci U S A. 118 (35 ):e2014781118. 10.1073/pnas.2014781118 34452993
23 Pollatos  O, Herbert  BM, Matthias  E, Schandry  R. 2007. Heart rate response after emotional picture presentation is modulated by interoceptive awareness. Int J Psychophysiol. 63 (1 ):117–124. 10.1016/j.ijpsycho.2006.09.003 17137662
24 Ba  D, Temereanca  S, Brown  EN. 2014. Algorithms for the analysis of ensemble neural spiking activity using simultaneous-event multivariate point-process models. Front Comput Neurosci. 8 :6. 10.3389/fncom.2014.00006 24575001
25 Souchet  AD, Philippe  S, Lourdeaux  D, Leroy  L. 2022. Measuring visual fatigue and cognitive load via eye tracking while learning with virtual reality head-mounted displays: a review. Int J Human–Comput Inter. 38 (9 ):801–824. 10.1080/10447318.2021.1976509
26 García-Cordero  I , et al. 2016. Feeling, learning from and being aware of inner states: interoceptive dimensions in neurodegeneration and stroke. Philos Trans R Soc B Biol Sci. 371 (1708 ):20160006. 10.1098/rstb.2016.0006
27 Amin  R, Faghih  RT. 2022. Physiological characterization of electrodermal activity enables scalable near real-time autonomic nervous system activation inference. PLoS Comput Biol. 18 (7 ):e1010275. 10.1371/journal.pcbi.1010275 35900988
28 Luck  SJ . 2014. An introduction to the event-related potential technique. MIT press.
29 Sur  S, Sinha  VK. 2009. Event-related potential: an overview. Ind Psychiatry J. 18 (1 ):70. 10.4103/0972-6748.57865 21234168
30 Rose  MF, Ahmad  KA, Thaller  C, Zoghbi  HY. 2009. Excitatory neurons of the proprioceptive, interoceptive, and arousal hindbrain networks share a developmental requirement for math1. Proc Natl Acad Sci U S A. 106 (52 ):22462–22467. 10.1073/pnas.0911579106 20080794
31 Galley  N, Betz  D, Biniossek  C. 2015. Fixation durations-why are they so highly variable?  Das Ende von Rational Choice? Zur Leistungsfähigkeit Rational-Choice-Theorie. 93 :83–106.
32 Dahl  MJ, Mather  M, Sander  MC, Werkle-Bergner  M. 2020. Noradrenergic responsiveness supports selective attention across the adult lifespan. J Neurosci. 40 (22 ):4372–4390. 10.1523/JNEUROSCI.0398-19.2020 32317388
33 Stuart  S , et al. 2019. Validation of a velocity-based algorithm to quantify saccades during walking and turning in mild traumatic brain injury and healthy controls. Physiol Meas. 40 (4 ):044006. 10.1088/1361-6579/ab159d 30943463
34 Nyström  M, Holmqvist  K. 2010. An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data. Behav Res Methods. 42 (1 ):188–204. 10.3758/BRM.42.1.188 20160299
35 Hamm  AO, Greenwald  MK, Bradley  MM, Lang  PJ. 1993. Emotional learning, hedonic change, and the startle probe. J Abnorm Psychol. 102 (3 ):453–465.10.1037/0021-843X.102.3.453 8408958
36 Wickramasuriya  DS, Qi  C, Faghih  RT. 2018. A state-space approach for detecting stress from electrodermal activity. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE. p. 3562–3567. 10.1109/EMBC.2018.8512928
37 Wiediartini , Ciptomulyono  U, Dewi  RS. 2023. Evaluation of physiological responses to mental workload in n-back and arithmetic tasks. Ergonomics. 67 (8 ):1121–1133. 10.1080/00140139.2023.2284677 37970874
38 Bach  DR, Flandin  G, Friston  KJ, Dolan  RJ. 2009. Time-series analysis for rapid event-related skin conductance responses. J Neurosci Methods. 184 (2 ):224–234. 10.1016/j.jneumeth.2009.08.005 19686778
39 Benedek  M, Kaernbach  C. 2010. A continuous measure of phasic electrodermal activity. J Neurosci Methods. 190 (1 ):80–91. 10.1016/j.jneumeth.2010.04.028 20451556
40 Benedek  M, Kaernbach  C. 2010. Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology. 47 (4 ):647–658. 10.1111/j.1469-8986.2009.00972.x 20230512
41 Greco  A, Valenza  G, Lanata  A, Scilingo  EP, Citi  L. 2015. cvxeda: a convex optimization approach to electrodermal activity processing. IEEE Trans Biomed Eng. 63 (4 ):797–804.26336110
42 Hernando-Gallego  F, Luengo  D, Artés-Rodríguez  A. 2017. Feature extraction of galvanic skin responses by nonnegative sparse deconvolution. IEEE J Biomed Health Inform. 22 (5 ):1385–1394. 10.1109/JBHI.2017.2780252 29990244
43 Bach  DR, Daunizeau  J, Friston  KJ, Dolan  RJ. 2010. Dynamic causal modelling of anticipatory skin conductance responses. Biol Psychol. 85 (1 ):163–170. 10.1016/j.biopsycho.2010.06.007 20599582
44 Sharma  K, Castellini  C, van den Broek  EL, Albu-Schaeffer  A, Schwenker  F. 2019. A dataset of continuous affect annotations and physiological signals for emotion analysis. Sci Data. 6 (1 ):196. 10.1038/s41597-019-0209-0 31597919
45 Esteves  F, Parra  C, Dimberg  U, Öhman  A. 1994. Nonconscious associative learning: pavlovian conditioning of skin conductance responses to masked fear-relevant facial stimuli. Psychophysiology. 31 (4 ):375–385.10690918
46 Fletcher-Watson  S, Hampton  S. 2018. The potential of eye-tracking as a sensitive measure of behavioural change in response to intervention. Sci Rep. 8 (1 ):14715. 10.1038/s41598-018-32444-9 30279422
47 Pagliaccio  D , et al. 2019. Cross-species convergence in pupillary response: understanding human anxiety via non-human primate amygdala lesion. Soc Cogn Affect Neurosci. 14 (6 ):591–599. 10.1093/scan/nsz041 31184751
48 Innocente  BP, Weingast  LT, George  R, Norrholm  SD. 2020. Psychophysiology of emotional responding in PTSD. In: Emotion in Posttraumatic Stress Disorder. Computers. California (USA): IEEE. p. 251–291. 10.1016/B978-0-12-816022-0.00009-0
49 Luna  B, Velanova  K, Geier  CF. 2008. Development of eye-movement control. Brain Cogn. 68 (3 ):293–308. 10.1016/j.bandc.2008.08.019 18938009
50 Lui  LL, Bourne  JA, Rosa  MGP. 2013. Relationship between size summation properties, contrast sensitivity and response latency in the dorsomedial and middle temporal areas of the primate extrastriate cortex. PLoS One. 8 (6 ):e68276. 10.1371/journal.pone.0068276 23840842
51 Fekri Azgomi  H, Branco  LRF, Amin  MR, Khazaei  S, Faghih  RT. 2023. Regulation of brain cognitive states through auditory, gustatory, and olfactory stimulation with wearable monitoring. Sci Rep. 13 (1 ):12399. 10.1038/s41598-023-37829-z 37553409
52 Foroughi  CK, Sibley  C, Coyne  JT. 2017. Pupil size as a measure of within-task learning. Psychophysiology. 54 (10 ):1436–1443.28593652
53 Dai  W, Selesnick  I, Rizzo  J-R, Rucker  J, Hudson  T. 2017. A nonlinear generalization of the Savitzky-Golay filter and the quantitative analysis of saccades. J Vis. 17 (9 ):10–10. 10.1167/17.9.10
54 Hamm  AO, Cuthbert  BN, Globisch  J, Vaitl  D. 1997. Fear and the startle reflex: blink modulation and autonomic response patterns in animal and mutilation fearful subjects. Psychophysiology. 34 (1 ):97–107. 10.1111/j.1469-8986.1997.tb02420.x 9009813
55 Bradley  M, Lang  PJ. 2007. Emotion and motivation. In: Cacioppo JT, Tassinary LG, Berntson G, editors. Handbook of psychophysiology. New York (US): Cambridge University Press.
56 Cacioppo  JT, Tassinary  LG, Berntson  G. 2007. Handbook of psychophysiology. Cambridge University Press.
57 Reddy  R, Khazaei  S, Faghih  RT. 2023. A point process approach for tracking valence using a respiration belt. In: 2023 45th Annual International Conference of the IEEE Engineering in Medicine and Biology Society EMBC. Sydney (Australia): IEEE. 10.1109/EMBC40787.2023.10339976
58 Aminihajibashi  S, Hagen  T, Andreassen  OA, Laeng  B, Espeseth  T. 2020. The effects of cognitive abilities and task demands on tonic and phasic pupil sizes. Biol Psychol. 156 :107945. 10.1016/j.biopsycho.2020.107945 32889001
59 Amin  MR, Wickramasuriya  DS, Faghih  RT. 2022. A wearable exam stress dataset for predicting grades using physiological signals. In: 2022 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT). IEEE. p. 30–36.
60 Amin  MR, Faghih  RT. 2020. Identification of sympathetic nervous system activation from skin conductance: a sparse decomposition approach with physiological priors. IEEE Trans Biomed Eng. 68 (5 ):1726–1736.
61 Amin  MR, Faghih  RT. 2019. Tonic and phasic decomposition of skin conductance data: a generalized-cross-validation-based block coordinate descent approach. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE. p. 745–749.
62 Amin  MDR, Faghih  RT. 2019. Robust inference of autonomic nervous system activation using skin conductance measurements: a multi-channel sparse system identification approach. IEEE Access. 7 :173419–173437.
63 Amin  MR, Faghih  RT. 2019. Sparse deconvolution of electrodermal activity via continuous-time system identification. IEEE Trans Biomed Eng. 66 (9 ):2585–2595.30629490
64 Amin  MR, Faghih  RT. 2018. Inferring autonomic nervous system stimulation from hand and foot skin conductance measurements. In: 52th Asilomar Conference on Signals, Systems and Computers. California (USA): IEEE..
65 Amin  MR, Tahir  M, Faghih  RT. 2021. An investigation of music impacts on cognitive performance and arousal in the presence of Yerkes-Dodson law. In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE.
66 Alam  S, Amin  MR, Faghih  RT. 2023. Sparse multichannel decomposition of electrodermal activity with physiological priors. IEEE Open J Eng Med Biol. 4 :234–250.38196978
67 Wickramasuriya  DS, Crofford  LJ, Widge  AS, Faghih  RT. 2022. Hybrid decoders for marked point process observations and external influences. IEEE Trans Biomed Eng. 70 :343–353.35839187
68 Azgomi  HF, Faghih  RT. 2022. Enhancement of closed-loop cognitive stress regulation using supervised control architectures. IEEE Open J Eng Med Biol. 3 :7–17.35399789
69 Wickramasuriya  DS, Faghih  RT. 2019. A bayesian filtering approach for tracking arousal from binary and continuous skin conductance features. IEEE Trans Biomed Eng. 67 (6 ):1749–1760.31603767
70 Wickramasuriya  DS, Faghih  RT. 2019. A novel filter for tracking real-world cognitive stress using multi-time-scale point process observations. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE. p. 599–602.
71 Wickramasuriya  DS, Faghih  RT. 2024. Bayesian filter design for computational medicine: a state-space estimation framework. Springer Nature.
72 Weng  HY , et al. 2021. Interventions and manipulations of interoception. Trends Neurosci. 44 (1 ):52–62.33378657
73 Suksasilp  C, Garfinkel  SN. 2022. Towards a comprehensive assessment of interoception in a multi-dimensional framework. Biol Psychol. 168 :108262.35026353
74 Melinšcak Y. Xia  F, Bach  DR. 2020. PsPM-VIS: SCR, ECG, respiration and eyetracker measurements in a delay fear conditioning task with visual CS and electrical US: Zenodo repository. 10.5281/zenodo.4290767
75 Kret  ME, Sjak-Shie  EE. 2019. Preprocessing pupil size data: guidelines and code. Behav Res Methods. 51 (3 ):1336–1342. 10.3758/s13428-018-1075-y 29992408
76 Stemerding  LE, van Ast  VA, Gerlicher  AMV, Kindt  M. 2022. Pupil dilation and skin conductance as measures of prediction error in aversive learning. Behav Res Ther. 157 :104164. 10.1016/j.brat.2022.104164 35994954
77 Alekseevsky  D . 2022. Microsaccades, drifts, hopf bundle and neurogeometry. J Imaging. 8 (3 ):76. 10.3390/jimaging8030076 35324631
78 Dai  W, Selesnick  I, Rizzo  J-R, Rucker  J, Hudson  T. 2021. Detection of normal and slow saccades using implicit piecewise polynomial approximation. J Vis. 21 (6 ):8–8. 10.1167/jov.21.6.8
79 Smith  AC , et al. 2004. Dynamic analysis of learning in behavioral experiments. J Neurosci. 24 (2 ):447–461. 10.1523/JNEUROSCI.2908-03.2004 14724243
