
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

72358
10.1038/s41598-024-72358-3
Article
Neural signatures of motor imagery for a supernumerary thumb in VR: an EEG analysis
Alsuradi Haneen 14
Hong Joseph 1
Sarmadi Alireza 3
Volcic Robert 245
Salam Hanan 24
Atashzar S. Farokh 34
Khorrami Farshad 34
Eid Mohamad mohamad.eid@nyu.edu

14
1 https://ror.org/00e5k0821 grid.440573.1 0000 0004 1755 5934 Engineering Division, New York University Abu Dhabi, Abu Dhabi, UAE
2 https://ror.org/00e5k0821 grid.440573.1 0000 0004 1755 5934 Division of Science, New York University Abu Dhabi, Abu Dhabi, UAE
3 https://ror.org/0190ak572 grid.137628.9 0000 0004 1936 8753 Department of Electrical and Computer Engineering, New York University, New York, NY USA
4 https://ror.org/00e5k0821 grid.440573.1 0000 0004 1755 5934 Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE
5 https://ror.org/00e5k0821 grid.440573.1 0000 0004 1755 5934 Center for Brain and Health, New York University Abu Dhabi, Abu Dhabi, UAE
16 9 2024
16 9 2024
2024
14 2155820 5 2024
5 9 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Human movement augmentation is a rising field of research. A promising control strategy for augmented effectors involves utilizing electroencephalography through motor imagery (MI) functions. However, performing MI of a supernumerary effector is challenging, to which MI training is one potential solution. In this study, we investigate the validity of a virtual reality (VR) environment as a medium for eliciting MI neural activations for a supernumerary thumb. Specifically, we assess whether it is possible to induce a distinct neural signature for MI of a supernumerary thumb in VR. Twenty participants underwent a two-fold experiment in which they observed movements of natural and supernumerary thumbs, then engaged in MI of the observed movements. Spectral power and event related desynchronization (ERD) analyses at the group level showed that the MI signature associated with the supernumerary thumb was indeed distinct, significantly different from both the baseline and the MI signature associated with the natural thumb, while single-trial classification showed that it is distinguishable with a 78% and 69% classification accuracy, respectively. Furthermore, spectral power and ERD analyses at the group level showed that the MI signatures associated with directional movement of the supernumerary thumb, flexion and extension, were also significantly different, and single-trial classification demonstrated that these movements could be distinguished with 60% accuracy. Fine-tuning the models further increased the respective classification accuracies, indicating the potential presence of personalized features across subjects.

Subject terms

Electrical and electronic engineering
Sensorimotor processing
Biomedical engineering
Tamkeen, UAECG10 CG10 CG10 CG10 CG10 CG10 CG10 Alsuradi Haneen Hong Joseph Volcic Robert Salam Hanan Atashzar S. Farokh Khorrami Farshad Eid Mohamad issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

A rising field of research in human movement augmentation is that of brain-computer interface (BCI) systems, which bypass the need for physical action to control the supernumerary effector and allow communication between the brain and device1. EEG, in particular, is a popular technique used for BCI control due to its non-invasive nature and high temporal resolution2. This method can capture time-frequency domain features such as ERD in the alpha band (9–13 Hz) at the motor cortex, phenomena linked to motor execution (ME) and MI3, of which the latter is often utilized in BCI applications. Despite its popularity, one major limitation of EEG is its low spatial resolution, which hinders the reliability of control, sometimes to the point where it can negatively impact both the controllability and concurrency of the effector. Given this, many implementations of EEG-based augmentation by extension often rely on triggering movements using MI signatures of pre-existing actions, such as walking in a specific direction or grasping an object4–6, improving reliability of control at the expense of concurrency.

However, addressing this limitation directly and without compromise is a possibility that can be explored. The motor cortex has been found to possess the ability to represent additional natural effectors and their movements through brain plasticity without impeding motor functions7,8, showing the potential of a representation for supernumerary effector control. A recent field of research explores mapping the control of a supernumerary effector to a distinct MI signature. For example, Penaloza et al. proposed a BCI system that controlled an augmented third arm through MI associated with the supernumerary effector while the natural arms were simultaneously engaged with a ball-balancing task9,10, indicating a concurrency between supernumerary and natural effector control. In addition, a 4-week longitudinal study performed by Liu et al. provided further evidence towards the feasibility of creating a distinct MI function that could be mapped to the control of a supernumerary thumb, achieving a within-subject classification accuracy of 70% in distinguishing between MI and rest periods11. While both of these works involved MI of the supernumerary effector, it remains unclear whether the MI performed by individuals was strictly associated with the supernumerary effector itself. There is a lack of direct comparison between the neural signatures associated with the MI of movements of the supernumerary effector and those of the natural one, which is necessary to verify distinctiveness. Furthermore, these studies highlighted the need for more reliability in the induced neural activations, as MI performance and clarity varied across subjects such that classification could only be performed on an individual level (i.e., within-subject classification).

One potential solution to these limitations in reliability is MI training, a concept relatively novel to the fields of BCI and augmentation, but well-researched in the fields of rehabilitation and sports12–14. MI training methods use individual or combinations of different motor stimuli in a repetitive manner to enhance MI performance, often associated with improved motor performance15. These stimuli include ME, physically performing the task to be imagined; motor observation (MO), observing the task being performed by someone else; or MI, mentally rehearsing the task without physically performing it12. However, we find that there are differences that should be considered in MI training of a supernumerary effector, as opposed to MI training for movements that involve natural effectors.

For instance, literature on MI training suggests that ME be interleaved with MI trials for better performance12. However, it is difficult to ask an individual to perform the movement of an effector that is completely foreign, thus MO could be considered as a close substitute13. Additionally, given that first person MO and MI is reported to induce a greater sense of embodiment and may involve different parts of the brain16,17, mediums that allow a first person MO of an unfamiliar effector, such as virtual reality (VR) or extended reality (XR), could be considered as a training environment. Furthermore, though not as essential in rehabilitation and sports, ensuring that motor stimuli are performed sequentially rather than simultaneously is crucial from a BCI perspective. This is due to differences in the respective neural activations of MO and MI14,18, as well as the effects they are induced in close tandem15. Considering that the goal of the method is to control a supernumerary effector using motor imagery, performing MO and MI in parallel could create different neural signatures from isolated MI, hindering the quality of EEG data.

Thus, in this study, our objective is to examine the validity of VR as an environment in eliciting distinct neural signature for a supernumerary effector, in this case, a virtual supernumerary thumb, through a cross-sectional study, by creating an MI paradigm involving two separate (i.e., consecutive but not simultaneous) first-person motor stimulations, MO and MI. We take a systematic approach by a) confirming the presence of distinguishable ERD during the MI of the supernumerary thumb; b) verifying that there is a distinction between the MI associated with the supernumerary thumb and the natural thumb; and c) identifying directional control within the MI of the supernumerary thumb. We employ trial-averaged statistical analysis and deep learning classification at each of these steps as a mode of validation of the results on a group-level and single-trial basis, respectively. We employ leave-one-subject-out cross-validation, examining the ability to extract group-level MI features while also examining the potential presence of personalized features by fine-tuning the model on test-subject data15,19,20.

Through this study, we contribute to the field in several ways. We first validate the VR as a method to induce MI neural activation for supernumerary effectors, promoting it as a potential MI training environment. We also show the distinction of the neural signature associated with the MI od the supernumerary effector against that of a natural effector of similar nature, which has not yet been performed in previous literature to the best knowledge of the authors. We also investigate signatures associated with the MI of flexion and extension of the supernumerary effector, providing insight on directional control of an augmented, BCI-based effector. Furthermore, we perform these classifications on a group level, as opposed to only on an individual basis, exploring the possibility of generalized classification models that can be potentially personalized through fine-tuning.

Methods

Participants

Twenty subjects were recruited for this experiment, comprising 14 males and 6 females. Of these, 18 were undergraduate students aged between 18 and 25 years and 2 were individuals between 30 and 50 years. All participants were right-handed, performed tasks with their right hand, and had normal or corrected-to-normal vision. Exclusion criteria were being under 18, left-handed, or having a history of traumatic brain injury, neurological disorders, or muscle atrophy. Participants provided informed consent for their participation as per the Institutional Review Board (IRB) ethics, with the study being approved by the New York University Abu Dhabi’s IRB (HRPP-2023-60), in accordance with the Declaration of Helsinki. Participants received subsistence allowance for their participation. Four participants were excluded due to poor electrode connections or cap misalignment caused by VR headset movement.

EEG apparatus

EEG data were recorded at a 1 kHz sampling rate using a BrainAmps Standard amplifier (Brain Products GmbH, Germany) during the experimental sessions. The Brain Vision Recorder software (Version 1.21.020121) managed the data acquisition process and monitored electrode connectivity. We utilized 64 active Ag/AgCl electrodes with integrated readout circuitry for noise cancellation and signal amplification. Electrodes were positioned according to the international 10–20 system, ensuring the Cz electrode was at the head’s vertex. The ground electrode was positioned at FPz, with the online reference at FCz. To maintain signal integrity, electrode-scalp impedance was kept below 10 kΩ. Data from channels T7, T8, FT7, and FT8 were excluded due to electromagnetic interference from the VR headset’s battery. Additionally, channels FT9, FT10, TP9, and TP10 were omitted because of inconsistent contact with the scalp. Figure 1a shows the setup of the experimentFigure 1 Experimental setup and protocol. (a) A participant wearing an EEG cap and a VR headset while resting their forearm on a towel for comfort. The participant had access to a keypad for self-assessment of their motor imagery performance. (b) A VR scene showing the virtual hand with the supernumerary thumb attached. The supernumerary thumb was modeled using Blender software (Version 3.422). (c) The experimental protocol included three main activities: a questionnaire, the main experiment with ten sessions, and a post-experiment survey. Each session involved embodiment exercise followed by sixteen trials, where participants mentally replicated movements involving flexion or extension of their natural thumb or the supernumerary thumb. Participants assessed their motor imagery execution after each trial. ST: Supernumerary thumb.

Experimental protocol

The experiment consisted of three main activities: Kinesthetic visual imagery (KVI) questionnaire. The KVI questionnaire is a widely recognized tool for evaluating the visual and kinesthetic aspects of an individual’s motor imagery capabilities23. In this assessment, the experimenter performed and described five different movements, which are to be imagined by the participant in sequence. Each movement was to be imagined twice, once visually and the second kinesthetically. During evaluation, participants were assisted by the experimenter who helped them rate the clarity of their visual imagery and the vividness of kinesthetic sensations through a well-defined scale. Each modality, visual and kinesthetic, was scored out of a total of 25 points, distributed across the five movements. These are called as the visual MI score and the kinesthetic MI score.

Main experiment. Participants were seated comfortably while experimenters set up the EEG equipment and fitted the VR headset (Meta Quest Pro24); see Fig. 1. The main experiment comprised 10 sessions, where the initial session was designated for practice. The session started with an embodiment period consisting of two 30-seconds phases of embodiment exercises. In the first phase, participants could see both hands and were instructed to pop balloons appearing in the virtual space using their right hand. The second phase introduced a virtual supernumerary thumb for participants to use in popping balloons attached to their right hand. Participants were directed to use their supernumerary thumb in popping balloons. This is to elicit a sense of embodiment of the virtual hand and the supernumerary thumb. After the embodiment period, participants underwent sixteen trials of the task. During these trials, they were instructed to rest their right forearm on a comfortable towel with their right palm facing them (see Fig. 1a). In the VR environment, participants could see their hand equipped with a supernumerary thumb, as shown in Fig. 1b. Each trial commenced with the hand in a static state for 2 seconds, followed by a motor observation period of another 2 seconds, during which a pre-recorded movement of either the natural or the supernumerary thumb performing flexion or extension was shown. The virtual hand then disappeared and reemerged in the corresponding pose. A baseline period followed for 2 seconds, displaying a fixation cross aligned with the center and just in front of the palm. After the cross vanished, participants were asked to engage in motor imagery of the previously observed movement, and during this time the the virtual hand and supernumerary thumb were static. Subsequently, participants were asked to rate their success in performing the motor imagery using a keypad with their left hand (binary rating). Consequently, each participant completed 160 trials, with the initial 16 trials conducted during the practice session. These trials were evenly divided among the four conditions: flexion and extension movements for both the natural and supernumerary thumbs. Figure 1c illustrates the experimental timeline, detailing the sequence of events for an individual trial.

Post-experiment survey. After the completion of the 10 sessions, participants were asked to complete a questionnaire capturing their experience. Particularly, we asked participants to report on: (1) their ownership and agency of the virtual hand and (2) their ability to perform motor imagery for the natural and supernumerary thumbs. Those were assessed using a 5-point Likert scale.

EEG data analysis

Data processing

Pre-processing and analysis of the EEG data were performed offline using MATLAB release 2022a (MathWorks, United States) and EEGLAB toolbox (v14.1.2)25. EEG data were filtered using a passband Sinc FIR filter between 0.1 and 40 Hz. To mitigate artifacts from muscle activity, eye movements, and blinks, as well as to identify and exclude heavily affected channels, the Artifact Subspace Reconstruction (ASR)26 technique was used with the following settings: argflatline = 10, arghighpass = [0.025 0.075], argchannel = 0.8, argnoisy = 4, argburst = 20, argwindow = ‘off’. Applying ASR with a threshold (argburst) of 20 has been shown to effectively clean EEG data from ocular artifact while retaining the neural features of the data27. Subsequently, the Common Average Referencing (CAR)28 method was applied for re-referencing the EEG data, and the online reference channel, FCz, was retained. All trials that were rated as unsuccessful during the evaluation period were discarded. Approximately, 85% of the MI trials were retained, having been rated as successful, such that 86% and 84% of the natural and supernumerary thumb trials were retained, respectively.

The two periods of focus in this study are the motor observation period and the motor imagery period. EEG data were epoched relative to the onset of each period. This epoch window spanned from 2 seconds before to 3 seconds after the onset. We implemented time-frequency analysis by convolving the signal with a set of complex Morlet wavelets, defined as complex sine waves tapered by a Gaussian. The frequencies of the wavelets ranged from 1 to 40 Hz in 40 linearly spaced 1 Hz steps. The full-width at half-maximum (FWHM) ranged from about 380 ms to 100 ms with increasing wavelet peak frequency. This corresponds to the wavelets with a number of cycles that increases logarithmically from 2 to 10. The power of the transformed signals is then calculated by squaring their amplitude. For trial-averaged analysis, epochs of the same condition were then averaged for each participant. For spectral power analysis, we calculated the event related spectral perturbation (ERSP) and applied decibel conversion to normalize the power of the ERSPs with respect to the baseline following the formula: 10log10activitybaseline. Baseline normalization was applied to the data using a baseline window from -1.25 to -0.25 seconds29 relative to the onset for each period of interest.

As for the ERD calculation of the alpha band, we followed the methodology outlined by Pfurtscheller30,31. Specifically, we first bandpass filtered the epoched trials from 9 to 13 Hz. Next, we squared the amplitude samples to obtain power samples and then averaged these power samples over trials. Finally, we averaged a small number of consecutive power samples (i.e., 50 datapoints corresponding to 50 ms) to reduce variance and smoothen the signal. ERD was then calculated using the formula ERD=100×A-RR, where A is the activity during the epoch and R is the baseline.. Other frequency bands did not exhibit any neural patterns that are relevant to this study.

Region of interest (ROI) selection

We identified the ROIs for the trial-averaged statistical analysis to examine the desynchronization in the alpha band during the motor imagery and the motor observation periods using a data-driven approach. This approach is based on conditions averaging, rather than conditions difference, thus avoiding double-dipping32. For each of the MO and MI periods, both the ERSPs of both conditions (natural and supernumerary) were averaged together, and the mean alpha band power throughout the periods was compared to the mean of the baseline period. Regions were selected for further analysis if they exhibited a statistically significant reduction in alpha power relative to the baseline. Particularly, cortical regions that showed prominent discrepancy with respect to the baseline were identified. Activation within the identified regions was statistically compared to the baseline by pooling electrode data from each cortical region and performing statistical tests with respect to the baseline. The p-values are corrected with respect to the number of comparisons (i.e. identified regions). This selection method, grounded on averaged conditions, is orthogonal to condition differences thus avoiding circular inference32.

For the motor imagery period, three ROIs were identified: 1) Frontal: Fp1, Fp2, AF3, AF4, AFz; 2) Contralateral sensorimotor: P1, P3, P5, CP1, CP3, CP5, C1, C3, C5; and 3) Ipsilateral sensorimotor: P2, P4, P6, CP2, CP4, CP6, C2, C4, C6.

On the other hand, another set of ROIs were identified for the motor observation period: 1) Frontal: Fp1, Fp2, AF3, AF4, AFz; 2) Contralateral parieto-occipital: CP3, CP1, P7, P5, P3, P1, PO7, PO3, O1 and 3) Ipsilateral parieto-occipital: P4, P6, P8, PO4, PO8, O2. In accordance with our selection, the sensorimotor area has been associated with MI, while the parieto-occipital region has been reported to be involved during MO14,18. Figure 2 shows the ROIs selection based on the average-of-conditions for each period (i.e., MO, MI).Figure 2 Regions of interest. The region of interests were selected in a data-driven approach (MO: motor observation; MI: motor imagery; NT: natural thumb; ST: supernumerary thumb). The average power of all conditions was compared to the average baseline, defined as the period between -1.25 and -0.25 seconds before the onset. Highlighted regions are considered as ROIs as their power were significantly larger relative to the baseline (p<0.05, Wilcoxon sign-rank test).

Statistical analysis

For trial-averaged analysis, we averaged the channels within each identified ROI. As per the aim of this study, we have conducted three main comparisons regarding the ERD in the identified ROIs: (1) between the motor imagery of the supernumerary thumb and rest; (2) between the natural and supernumerary thumbs during the motor imagery period; and (3) between flexion and extension movements of the supernumerary thumb throughout the motor imagery period. We further explored differences in the neural signatures between the natural and supernumerary thumbs during the motor observation period.

All of these analyses were performed at the group level, with trials from individual participants averaged together. ERDs within the ROIs were segmented into 0.2 second time windows, each overlapping by 50%, and activation within each bin was averaged. To mitigate Type I error and adjust for multiple comparisons, the false discovery rate (FDR) method was applied to all p-values, such that for each analysis, a total of 60 comparisons (20 overlapping time-intervals × 3 ROIs) were accounted for in each analysis. This manuscript reports corrected p-values and identifies the time intervals where statistically significant differences were found. Surrogate tests are ideal for EEG data analysis because they make no assumptions about data distribution; therefore, for each time window, we use the bootstrap surrogate test provided by the EEGLAB toolbox25,33.

For the statistical analysis conducted to compare the improvement in classification accuracy after fine-tuning when performing single-trial classification, we used the paired t-test after confirming normality and corrected for multiple comparisons (5 comparisons,) using FDR. All statistical analyses were conducted using the Statistics toolbox in MATLAB 2022a and EEGLAB toolbox v.14.1.2.

Machine learning classification

We perform four binary classification tasks. The first task utilizes the EEG trials to distinguish between the supernumerary thumb MI and the baseline. The second task attempts to classify the natural thumb and the supernumerary thumb during the motor imagery period. In the third task, we aim to classify between flexion and extension of the supernumerary thumb during MI. The fourth and final task focuses on distinguishing supernumerary thumb MO from natural thumb MO.

The dataset consists of data from N=16 subjects. Each subject’s data consisted of approximately tr=30 trials per condition, with each condition defined as either flexion or extension of the natural or supernumerary thumbs (some trials were discarded due to unsuccessful motor imagery). The deep learning model takes as input a single trial Xi∈RC×T where C is the number of EEG channels (electrodes) and T is the number of time points in a trial. The model aims to map the input trial Xi to its corresponding label yi. Thus, the dataset consists of S={Xi,yi}i=1m, where C=56 (channels), T=2000 (data points; corresponding to a duration of 2 seconds with a sampling rate of 1 kHz), and m is the total number of trials.

In this paper, we use ATCNet34, an attention-based temporal convolutional network architecture for the classification tasks. This model consists of three main blocks: a convolutional (CV) block, an attention (AT) block, and a temporal convolutional (TC) block. The hyperparameters of the model are the same as in34, except for the dropout rate of the CV block which is 0.1 and number of sliding windows after the CV layer which is 9 in our case. The model was evaluated using the leave-one-subject-out strategy in which the data from all the subjects except one was used for training the model and the left out subject was utilized to evaluate the performance of the model. Leave-one-subject-out ensures that the model is able to learn the underlying features in the EEG data corresponding to the task that are independent from the subjects. The model was trained for 30 epochs with a batch size of 16. Binary Cross Entropy (BCE) loss was chosen for the loss function since the tasks are binary classifications. Adam optimizer was utilized with a learning rate of 0.00005 and a weight decay of 0.01. The model training was performed on an NVIDIA Quadro RTX 5000 GPU. For each of the analyses presented in the paper, the mean and standard deviation of the accuracy across all 16 subjects were calculated.

Results

The outcomes of the pre- and post-experiment questionnaires are analyzed to confirm the subjects’ ability to perform MI of the supernumerary thumb. Once confirmed, we conduct a step-by-step analysis of the EEG data. First, we examine if there exists a neural signature for the MI of the supernumerary thumb. If so, we determine whether this signature is distinct from that of the natural thumb. Next, we investigate the directional movement of the supernumerary thumb to see if there is a difference between the neural signatures of MI of flexion and MI of extension. Lastly, we analyze the neural activation during the motor observation period to identify any significant differences during passive observation of the movement of the supernumerary thumb versus the natural thumb. For each analysis, we employ trial-averaged statistical analysis and deep learning classification to validate the results on a group-level and single-trial basis, respectively.

Pre- and post-experiment questionnaires

Before the experiment, participants were asked to complete the KVI Questionnaire23 to provide a baseline of their MI abilities. Participants showed a decent MI ability, scoring an average of 22.4 (SD = 2.5) out of 25 for visual MI and 18.4 (SD = 4.3) out of 25 for kinesthetic MI.

After the experiment, participants were asked to complete a survey that evaluated their embodiment within the VR space, as well as their ability to perform MI for the natural thumb and the supernumerary thumb. On a scale of 5, participants reported a sense of ownership (M= 3.67, SD = 0.65) and agency (M = 3.95, SD = 0.69) towards their virtual hand. Participants rated highly their ability to perform MI for the natural thumb (M = 3.80, SD = 0.70) and MI for the supernumerary thumb (M = 3.90, SD = 0.64), with no significant differences between the two.

Neural analysis of motor imagery

Supernumerary thumb

The trial-averaged analysis shows that there exists a clear ERD pattern in the alpha band during the MI of the supernumerary thumb. Figure 3a shows the grand average topography map of the ERD during the motor imagery period, normalized with respect to the baseline. The figure shows a clear neural activation in the identified ROIs. The ERD in the three ROIs is significantly higher than their counterparts during the baseline period (i.e. zero ERD), as depicted in Fig. 3b. The ability to identify neural signatures of single-trial MI of the supernumerary thumb is confirmed through deep learning classification against baseline. We employed the ATCNet model34 for this classification task, as well as for the rest of the classification tasks in the remainder of this manuscript. Figure 3c shows the leave-one-subject-out classification accuracy, before and after fine-tuning. The average accuracy obtained is approximately 78.7% and 82.1%, for before and after fine-tuning, respectively, where 50% of the data of the test subject are used for fine-tuning, and only the remaining 50% were used in testing.Figure 3 MI of supernumerary thumb versus baseline. (a) The grand-averaged topography plot showing the alpha ERD during the MI of the supernumerary thumb. (b) Averaged alpha ERD compared to baseline for each ROI. p < 0.001(***), bootstrap test, FDR correction, (N = 16 samples). (c) Classification accuracy of MI of the supernumerary thumb versus baseline under leave-one-subject-out cross validation before and after fine-tuning with 50% of the test-subject data. The average accuracy is presented as a dashed blue (without fine-tuning) or red (with fine-tuning) line. ST: Supernumerary thumb.

Figure 4 Natural thumb versus supernumerary thumb during the motor imagery period. (a) ERSP heatmaps for the three ROIs for the motor imagery period. (b) Time course of scalp topographies showing the alpha ERD with a resolution of 0.2 seconds averaged over all participants. (c) Time course of the grand-averaged alpha ERD for the three ROIs identified for the motor imagery period. The shaded area marks the time interval where a statistically significant difference in ERD was observed between the natural and supernumerary thumb motor imagery (p<0.05, bootstrap test, FDR correction); trange= [0.5 to 0.9 seconds]). NT: Natural thumb, ST: Supernumerary thumb.

Figure 5 Classification of the natural thumb versus supernumerary thumb during the motor imagery period. (a) Classification accuracy of MI of the supernumerary thumb versus the MI of the natural thumb under leave-one-subject-out cross validation before and after fine-tuning with 50% of the test-subject data. The average accuracy is presented as a dashed blue (without fine-tuning) or red (with fine-tuning) line. (b) The change in the average accuracy with the increase of the percentage of the data used from the test subject for fine-tuning. The statistical significance of the change is assessed; p< 0.05(*), paired t-test, FDR correction.

Supernumerary thumb versus natural thumb

We compared the neural signatures of the natural and supernumerary thumbs during MI, combining effector flexion or extension. Figures 4a and b show the ERSP heatmaps for the three identified ROIs and the topographic maps in the alpha band, respectively. Through the trial-averaged analysis, it can be observed, as hypothesized, that the ERD is centered in the alpha band in the contralateral and ipsilateral sensorimotor areas. Furthermore, a statistically significant difference is observed in the ipsilateral sensorimotor region, where the supernumerary thumb exhibited a more pronounced ERD between 0.5 and 0.9 seconds from the onset of MI, shown in Fig. 4c (p<0.05, bootstrap test, FDR corrected).

From a single-trial perspective, detectable differences exist between MI activations of the natural and supernumerary thumbs. This is confirmed by the leave-one-subject-out classification results, shown in Fig. 5a. The average classification accuracy is 69.1% and 72.2% before and and after fine-tuning, respectively, where 50% of the data of the test subject are used for fine-tuning, and only the remaining 50% were used in testing. In Fig. 5b, a slight increase in the average classification accuracy is evident as more of the test-subject data is employed for fine-tuning. Notably, this increase in average accuracy becomes significantly greater than that of no fine-tuning once 50% of the test-subject data are utilized.Figure 6 Flexion versus extension movements for the supernumerary thumb during the motor imagery period. (a) ERSP heatmaps for the three ROIs for the motor imagery period. (b) Time course of scalp topographies showing alpha ERD with a resolution of 0.2 seconds averaged over all participants. (c) Time course of the grand-averaged alpha ERD for the three ROIs identified for the motor imagery period. The shaded area marks the time interval where a statistically significant difference in ERD was observed between the flexion and extension of the supernumerary thumb motor imagery (p<0.05, bootstrap test, FDR correction); trange= [0.4–0.7 seconds and 0.8–1.2 seconds]).

Flexion versus extension of the supernumerary thumb

We further investigated differences in the neural signature between the MI of flexion and extension, motivated by the interest in directional control of the supernumerary thumb. Through the trial-averaged analysis, it can be observed that the MIs of both flexion and extension exhibit an ERD centered in the alpha band for the sensorimotor areas (Fig. 6a). Additionally, the MI corresponding to extension seems to introduce less localized ERD patterns across the scalp, particularly between 0.4 and 1.4 seconds from the onset; see Fig. 6b. Furthermore, Fig. 6c shows that, indeed, a statistically significant difference in the ERD between flexion and extension was found in the frontal region between 0.4 and 0.7 seconds and between 0.8 and 1.2 seconds from the MI onset (p<0.05, bootstrap test, FDR corrected). This can be visually detected in Fig.6a, where those differences appear to exist in lower frequency bands as well.

The ability to differentiate between the MI of flexion from extension for the supernumerary thumb through a single-trial EEG data has also been attempted. The accuracy of the leave-one-subject-out classification is presented in Fig. 7a. The average classification accuracy is 60.1% and 65.8%, for before and after fine-tuning, respectively, where the impact of fine-tuning appears to be more pronounced for some subjects than for others.

Figure 7b shows the increase in the average classification accuracy as more test subject data is used for fine-tuning. The average accuracy becomes significantly greater than the average accuracy without fine-tuning after utilizing at least 40% of the test-subject data.Figure 7 Classification of flexion versus extension movements for the supernumerary thumb during the motor imagery period. (a) Classification accuracy of MI of flexion versus extension for the supernumerary thumb under leave-one-subject-out cross validation before and after fine-tuning with 50% of the test-subject data. The average accuracy is presented as a dashed blue (without fine-tuning) or red (with fine-tuning) line. (b) The change in the average accuracy with the increase of the percentage of the data used from the test subject for fine-tuning. The statistical significance of the change is assessed; p < 0.05(*), paired t-test, FDR correction.

Neural analysis of motor observation

Figure 8 Natural thumb versus supernumerary thumb during the motor observation period. (a) ERSP heatmaps for the three ROIs for the motor observation period. (b) Time course of scalp topographies showing alpha ERD with a resolution of 0.2 seconds averaged over all participants. (c) Time course of the grand-averaged alpha ERD for the three ROIs identified for the motor observation period. The shaded area marks the time interval where a statistically significant difference in alpha power was observed between the natural and supernumerary thumb motor observation (p<0.05, bootstrap test, FDR correction); trange= [0.3–0.6 seconds]). NT: Natural thumb, ST: Supernumerary thumb.

Figure 9 Classification of natural thumb versus supernumerary thumb during the motor observation period. (a) Classification accuracy of the supernumerary and the natural thumbs during MO under leave-one-subject-out cross validation before and after fine-tuning with 50% of the test-subject data. The average accuracy is presented as a dashed blue (without fine-tuning) or red (with fine-tuning) line. (b) The change in the average accuracy with the increase of the percentage of the data used from the test subject for fine-tuning. The statistical significance of the change is assessed; p< 0.05(*), paired t-test, FDR correction.

As mentioned above, the motor observation period was incorporated into the experimental paradigm to guide and improve MI quality for the supernumerary thumb. A difference in alpha ERD between the natural thumb and the supernumerary thumb during MO indicates that the visual processing of both effectors is distinct at the neural level.

Upon examining the ERD across the three ROIs during MO (see Fig. 2) and comparing the natural and supernumerary thumbs, we observed a more pronounced ERD in the frontal region for the supernumerary thumb. Figure 8a shows the ERSP heatmaps of the three ROIs, showing a clear and continuous desynchronization pattern during the motor observation period. Figure 8b depicts the time-course scalp topographies for the alpha ERD of both the natural and supernumerary thumbs during the motor observation period. The ERD of the alpha band at the frontal region is found to be significantly lower for the supernumerary thumb between 0.3 and 0.6 seconds from the MO onset, compared to the natural thumb (p< 0.05, bootstrap test, FDR corrected), as shown in Fig. 8c. This difference in the frontal region is visible from the time-course of scalp topographies (third time bin) shown in Fig. 8b. At the single-trial level, the deep learning model was able to classify the MO of the two thumbs apart with an average accuracy of 71.8% and 76.1%, before and after fine-tuning, respectively; see Fig. 9a. The average accuracy becomes significantly greater than the average accuracy without fine-tuning after utilizing at least 30% of the test-subject data (Fig. 9b).

Discussion

Through this study, we aimed to evaluate the validity of VR as an environment for eliciting neural synergies associated with a supernumerary thumb and exploring its potential for MI training, motivated by the benefits VR offers in terms of ease of use, availability, and reduced MI training cost. The results showed that individuals in the VR environment were not only able to reliably perform MI for the supernumerary thumb during the MI period, but also able to do so in a manner that could distinguish the supernumerary thumb from the natural one. The single-trial classification results are also promising, indicating that although there may be a certain degree of personal features involved in the MI process, there may also be a degree of shared features that could be further explored.

During the MI period, the trial-averaged difference between the supernumerary and natural thumbs was observed in the ipsilateral sensorimotor ROI, where the supernumerary thumb exhibited a statistically more pronounced bilateral ERD activation, compared to predominantly contralateral ERD activation for the natural thumb. Bilateral activation during unilateral MI is observed with movements of increased complexity35; a movement performed with a supernumerary effector is inherently more complex, and this may explain this difference. Another point of interest is the more widespread distribution of the ERD pattern across the scalp for the supernumerary thumb, consistent with findings from previous studies11. The MI of a supernumerary hand in VR has been shown to generate a widespread ERD in the alpha band across the scalp36, and previous studies have demonstrated that MI of an unfamiliar action recruits more brain resources and induces a stronger response than simpler or familiar actions37. Another hypothesis for this widespread ERD pattern is that the supernumerary thumb might be perceived as a tool rather than an owned body part. It has been reported that executing a task with a tool activates other brain regions than the motor cortex, such as the ipsilateral parietal region, more extensively than performing the same task with the hand38–40. The question of whether the extensive ERD pattern results from the supernumerary thumb being perceived as a tool, or due to the unfamiliarity of the effector, remains open and requires further investigation.

A distinction was also found in the neural signatures associated with the MI of flexion and extension of the supernumerary thumb. Adding a directional control to a supernumerary effector is pivotal, as it allows for more complex movements and better usage of the effector. Despite the relatively low classification accuracy at the single-trial level, the observed neural distinction between flexion and extension of the supernumerary thumb hints at the potential for developing more discernible neural activations through longitudinal training. Interestingly, this difference was not observed during the MI of the natural thumb.

We have also investigated the neural differences between the supernumerary and the natural thumbs during the MO period. The single-trial classification results (Fig. 9a) show a clear distinction between the two signatures. Furthermore, a significant difference in the alpha ERD is observed in the frontal region at the trial-averaged level. This indicates that the VR-based MO period showed a differentiation in the neural signature during the passive observation of the movement of the natural versus the supernumerary effector, possibly leading to a distinction between the two in the subsequent motor imagery period. This finding is also neural evidence of previous literature, reporting that providing realistic visual feedback that is designed to induce a sense of embodiment41,42, or a VR embodiment, assists in MI training43–45. Others reported the effectiveness of MO in promoting early learning of new and complex tasks46.

Despite both MO and MI periods eliciting a clear ERD, each had its own associated region: MO activated the parieto-occipital area, while MI activated the sensorimotor cortex. In a previous study by Liu et al.11, subjects were instructed to perform MI of bending the supernumerary thumb while simultaneously engaging in MO of the action displayed on a screen. Trials that were labeled as low performance in regards to MI were found to exhibit a greater ERD activation in the parieto-occipital region. Our results explain this observation, highlighting that in those trials, subjects could have just performed MO disregarding MI. This finding provides evidence for the need to separate motor stimulation tasks (MO and MI) for the quality of EEG data, particularly in the context of BCI and deep learning classification. Furthermore, despite observing a desynchronization in the beta band during both MI and MO for the different conditions as can be seen from the ERSP heatmaps, no statistically significant differences were observed across the analyzed conditions. This could be explained by the different functional roles that each of the frequency bands play during MI and MO47.

Our results also show the existence of neural activity in the frontal brain region. This is known to be associated with higher-order cognitive functions such as abstraction, and attention48. We hypothesize the activation of the frontal region is due to the two levels of abstraction involved. The first is caused by the VR environment where an ERD was observed for both, the natural and supernumerary thumb in the frontal region. The second layer of abstraction is for the supernumerary thumb being recognized as an artificial effector; the ERD in the frontal ROI was more pronounced for the supernumerary thumb during MO. This observation aligns with the findings by Penaloza et al.9 and Liu et al.11, where they reported that MI of a supernumerary effector (arm and thumb, respectively) in physical space influenced not just the sensorimotor areas but also the frontal area. The frontal ROI is a region that is susceptible to EOG artifacts. EOG artifacts generally appear in the delta band (extending to the theta band as well) as an increase in power in those bands with respect to non-contaminated EEG. However, the observed effects in our manuscript are desynchronizations in the alpha band, indicating a decrease in power, which is not characteristic of EOG artifacts49.

The trial-averaged statistical analysis presented in this study was complemented by single-trial deep learning classification using the leave-one-subject-out cross-validation method. The average classification accuracies obtained significantly surpass the chance level, yet with notable variability across subjects, suggesting diverse capabilities in performing MI for a supernumerary thumb. These findings demonstrate the presence of generalized MI features for the supernumerary thumb across the population, as evidenced by the model’s ability to generalize to unseen test-subject data. Additionally, fine-tuning the model on a subset of test-subject data was observed to enhance average classification accuracy. The slight increase in classification accuracy observed for most subjects after fine-tuning could be due to merely including more data for training, but possibly due to the presence of personalized neural features. Further validation with additional control experiments are required to conclusively attribute the improvements to personalized features.

It is also important to acknowledge the cross-sectional nature of this study. The study aims to investigate whether distinct MI neural signatures for a supernumerary effector can be elicited in a VR space using a novel paradigm involving sequential motor tasks, with the motivation of MI training. The results confirm that this is indeed possible on a cross-sectional level, meaning that the overall protocol was able to elicit these neural activations from participants who had no prior experience with the supernumerary effector. In this sense, participants seemed to have learned a new set of movements, those associated with the supernumerary thumb, and possibly could have improved in MI performance of these skills. However, given this is not a longitudinal study, it is difficult to make conclusions about the effectiveness of the paradigm in MI training, as well as the individual or combined effects of the MO and MI tasks. A longitudinal study would allow further understanding and evaluation of the paradigm and its components from an MI training perspective.

Similarly, the results confirm that VR is a valid medium through which the distinct neural signature of a supernumerary effector can be elicited, but do not evaluate its performance against the physical space and the presence of a physical supernumerary effector. The goal of the study was to investigate the feasibility of using VR with the proposed paradigm, given the aforementioned considerations and other possible benefits. For example, VR could enable the customization of the virtual supernumerary effector without the need to construct a new device over every iteration and, in the case of a larger or delicate effector, could ensure the safety of both the individual and the device. Thus, VR could be low-cost alternative for the development and training environment for a supernumerary effector. However, given the disparity between the virtual and physical space, further research would be needed to understand and evaluate the benefits that VR can offer, as well as how additional modalities such as haptic feedback could help bridge the gap and enhance the quality of MI50–52.

Collectively, this paper demonstrates the potential of VR as a platform for eliciting distinct MI signatures for supernumerary effectors, such as a supernumerary thumb, supported by single-trial and trial-averaged evidence. Future studies will focus on exploring the transition from VR to physical space following longitudinal training in VR, as well as investigating the effects of the proposed MI paradigm and its components in MI training. Furthermore, another interesting endeavor is to investigate whether, following longitudinal training, the activation associated with the supernumerary effector’s MI localizes, and whether this localization persists, thereby demonstrating either a short-term plasticity or a lasting adaptation.

Supplementary Information

Supplementary Information.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-024-72358-3.

Acknowledgements

The authors acknowledge Mbebo Nonna for his modeling and VR development efforts. Additionally, this work is supported in part by the NYUAD Center for Artificial Intelligence and Robotics, funded by Tamkeen under the NYUAD Research Institute Award CG010. RV was partially supported by the NYUAD Center for Brain and Health, funded by Tamkeen under the NYUAD Research Institute Award CG012.

Author contributions

M.E. led the study and, along with H.A., R.V., H.S., S.A., and F.K., conceived and developed the protocol. H.A. conducted the experiment. H.A. and J.H. analyzed the data, with H.A. and A.S. performing the machine learning analysis. All authors contributed to discussing the results, writing, and reviewing the manuscript.

Data availibility

EEG data is uploaded to OSF platform and the embargo will be lifted upon acceptance https://osf.io/xrgvy/.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Eden J Principles of human movement augmentation and the challenges in making it a reality Nat. Commun. 2022 13 1345 10.1038/s41467-022-28725-7 35292665
Eden, J. et al. Principles of human movement augmentation and the challenges in making it a reality. Nat. Commun. 13, 1345 (2022).35292665 10.1038/s41467-022-28725-7
2. Alsuradi H Park W Eid M Eeg-based neurohaptics research: A literature review IEEE Access 2020 8 49313 49328 10.1109/ACCESS.2020.2979855
Alsuradi, H., Park, W. & Eid, M. Eeg-based neurohaptics research: A literature review. IEEE Access 8, 49313–49328 (2020).10.1109/ACCESS.2020.2979855
3. Pfurtscheller G Spatiotemporal erd/ers patterns during voluntary movement and motor imagery Suppl. Clin. Neurophysiol. 2000 53 196 198 10.1016/S1567-424X(09)70157-6 12740996
Pfurtscheller, G. Spatiotemporal erd/ers patterns during voluntary movement and motor imagery. Suppl. Clin. Neurophysiol. 53, 196–198 (2000).12740996 10.1016/S1567-424X(09)70157-6
4. Tang Z Wearable supernumerary robotic limb system using a hybrid control approach based on motor imagery and object detection IEEE Trans. Neural Syst. Rehabil. Eng. 2022 30 1298 1309 10.1109/TNSRE.2022.3172974 35511846
Tang, Z. et al. Wearable supernumerary robotic limb system using a hybrid control approach based on motor imagery and object detection. IEEE Trans. Neural Syst. Rehabil. Eng. 30, 1298–1309 (2022).35511846 10.1109/TNSRE.2022.3172974
5. Meng J Noninvasive electroencephalogram based control of a robotic arm for reach and grasp tasks Sci. Rep. 2016 6 38565 10.1038/srep38565 27966546
Meng, J. et al. Noninvasive electroencephalogram based control of a robotic arm for reach and grasp tasks. Sci. Rep. 6, 38565 (2016).27966546 10.1038/srep38565
6. Lee Kyuhwa Liu Dong Perroud Laetitia Chavarriaga Ricardo Millán José del R. A brain-controlled exoskeleton with cascaded event-related desynchronization classifiers Robot. Autonom. Syst. 2017 90 15 23 10.1016/j.robot.2016.10.005
Lee, Kyuhwa, Liu, Dong, Perroud, Laetitia, Chavarriaga, Ricardo & Millán, José del R.. A brain-controlled exoskeleton with cascaded event-related desynchronization classifiers. Robot. Autonom. Syst. 90, 15–23. 10.1016/j.robot.2016.10.005 (2017).10.1016/j.robot.2016.10.005
7. Mehring C Augmented manipulation ability in humans with six-fingered hands Nat. Commun. 2019 10 2401 10.1038/s41467-019-10306-w 31160580
Mehring, C. et al. Augmented manipulation ability in humans with six-fingered hands. Nat. Commun. 10, 2401 (2019).31160580 10.1038/s41467-019-10306-w
8. Kieliba P Clode D Maimon-Mor RO Makin TR Robotic hand augmentation drives changes in neural body representation Sci. Robot. 2021 6 eabd7935 10.1126/scirobotics.abd7935 34043536
Kieliba, P., Clode, D., Maimon-Mor, R. O. & Makin, T. R. Robotic hand augmentation drives changes in neural body representation. Sci. Robot. 6, eabd7935 (2021).34043536 10.1126/scirobotics.abd7935
9. Penaloza CI Nishio S BMI control of a third arm for multitasking Sci. Robot. 2018 3 eaat1228 10.1126/scirobotics.aat1228 33141729
Penaloza, C. I. & Nishio, S. BMI control of a third arm for multitasking. Sci. Robot. 3, eaat1228 (2018).33141729 10.1126/scirobotics.aat1228
10. Penaloza, C., Hernandez-Carmona, D. & Nishio, S. Towards intelligent brain-controlled body augmentation robotic limbs. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 1011–1015 (IEEE, 2018).
11. Liu Y Wang Z Huang S Wang W Ming D EEG characteristic investigation of the sixth-finger motor imagery and optimal channel selection for classification J. Neural Eng. 2022 19 016001 10.1088/1741-2552/ac49a6
Liu, Y., Wang, Z., Huang, S., Wang, W. & Ming, D. EEG characteristic investigation of the sixth-finger motor imagery and optimal channel selection for classification. J. Neural Eng. 19, 016001 (2022).10.1088/1741-2552/ac49a6
12. Ladda AM Lebon F Lotze M Using motor imagery practice for improving motor performance-a review Brain Cogn. 2021 150 105705 10.1016/j.bandc.2021.105705 33652364
Ladda, A. M., Lebon, F. & Lotze, M. Using motor imagery practice for improving motor performance-a review. Brain Cogn. 150, 105705 (2021).33652364 10.1016/j.bandc.2021.105705
13. Mulder T Motor imagery and action observation: Cognitive tools for rehabilitation J. Neural Transm. 2007 114 1265 1278 10.1007/s00702-007-0763-z 17579805
Mulder, T. Motor imagery and action observation: Cognitive tools for rehabilitation. J. Neural Transm. 114, 1265–1278 (2007).17579805 10.1007/s00702-007-0763-z
14. Mizuguchi N Kanosue K Changes in brain activity during action observation and motor imagery: Their relationship with motor learning Prog. Brain Res. 2017 234 189 204 10.1016/bs.pbr.2017.08.008 29031463
Mizuguchi, N. & Kanosue, K. Changes in brain activity during action observation and motor imagery: Their relationship with motor learning. Prog. Brain Res. 234, 189–204 (2017).29031463 10.1016/bs.pbr.2017.08.008
15. Eaves DL Riach M Holmes PS Wright DJ Motor imagery during action observation: A brief review of evidence, theory and future research opportunities Front. Neurosci. 2016 10 514 10.3389/fnins.2016.00514 27917103
Eaves, D. L., Riach, M., Holmes, P. S. & Wright, D. J. Motor imagery during action observation: A brief review of evidence, theory and future research opportunities. Front. Neurosci. 10, 514 (2016).27917103 10.3389/fnins.2016.00514
16. Lorey B The embodied nature of motor imagery: The influence of posture and perspective Exp. Brain Res. 2009 194 233 243 10.1007/s00221-008-1693-1 19139856
Lorey, B. et al. The embodied nature of motor imagery: The influence of posture and perspective. Exp. Brain Res. 194, 233–243 (2009).19139856 10.1007/s00221-008-1693-1
17. Ruby P Decety J Effect of subjective perspective taking during simulation of action: A pet investigation of agency Nat. Neurosci. 2001 4 546 550 10.1038/87510 11319565
Ruby, P. & Decety, J. Effect of subjective perspective taking during simulation of action: A pet investigation of agency. Nat. Neurosci. 4, 546–550 (2001).11319565 10.1038/87510
18. Neuper C Scherer R Reiner M Pfurtscheller G Imagery of motor actions: Differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG Cogn. Brain Res. 2005 25 668 677 10.1016/j.cogbrainres.2005.08.014
Neuper, C., Scherer, R., Reiner, M. & Pfurtscheller, G. Imagery of motor actions: Differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. Cogn. Brain Res. 25, 668–677 (2005).10.1016/j.cogbrainres.2005.08.014
19. Niedermeyer E The normal EEG of the waking adult Electroencephalogr. Basic Princ. Clin. Appl. Related Fields 2005 167 155 164
Niedermeyer, E. et al. The normal EEG of the waking adult. Electroencephalogr. Basic Princ. Clin. Appl. Related Fields 167, 155–164 (2005).
20. Ono Y Wada K Kurata M Seki N Enhancement of motor-imagery ability via combined action observation and motor-imagery training with proprioceptive neurofeedback Neuropsychologia 2018 114 134 142 10.1016/j.neuropsychologia.2018.04.016 29698736
Ono, Y., Wada, K., Kurata, M. & Seki, N. Enhancement of motor-imagery ability via combined action observation and motor-imagery training with proprioceptive neurofeedback. Neuropsychologia 114, 134–142 (2018).29698736 10.1016/j.neuropsychologia.2018.04.016
21. Brainproducts. BrainAmp Standard. https://brainvision.com/products/brainamp-standard/ (2023). Online; Accessed 25 July 2024.
22. Blender. Blender Software. https://download.blender.org/release/Blender3.4/ (2023). Online; Accessed 25 July 2024.
23. Malouin F The kinesthetic and visual imagery questionnaire (kviq) for assessing motor imagery in persons with physical disabilities: A reliability and construct validity study J. Neurol. Phys. Ther. 2007 31 20 29 10.1097/01.NPT.0000260567.24122.64 17419886
Malouin, F. et al. The kinesthetic and visual imagery questionnaire (kviq) for assessing motor imagery in persons with physical disabilities: A reliability and construct validity study. J. Neurol. Phys. Ther. 31, 20–29 (2007).17419886 10.1097/01.NPT.0000260567.24122.64
24. Meta. Meta Quest Pro. https://www.meta.com/quest/quest-pro/ (2023). Online; Accessed 25 July 2024.
25. Delorme A Makeig S EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis J. Neurosci. Methods 2004 134 9 21 10.1016/j.jneumeth.2003.10.009 15102499
Delorme, A. & Makeig, S. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134, 9–21 (2004).15102499 10.1016/j.jneumeth.2003.10.009
26. Kothe, C. A. E. & Jung, T.-P. Artifact removal techniques with signal reconstruction (2016). US Patent App. 14/895,440.
27. Chang, C.-Y., Hsu, S.-H., Pion-Tonachini, L. & Jung, T.-P. Evaluation of artifact subspace reconstruction for automatic eeg artifact removal. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 1242–1245 (IEEE, 2018).
28. Lakshmi MR Prasad T Prakash DVC Survey on EEG signal processing methods Int. J. Adv. Res. Comput. Sci. Softw. Eng. 2014 4 84 91
Lakshmi, M. R., Prasad, T. & Prakash, D. V. C. Survey on EEG signal processing methods. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 4, 84–91 (2014).
29. Keil A Recommendations and publication guidelines for studies using frequency domain and time-frequency domain analyses of neural time series Psychophysiology 2022 59 e14052 10.1111/psyp.14052 35398913
Keil, A. et al. Recommendations and publication guidelines for studies using frequency domain and time-frequency domain analyses of neural time series. Psychophysiology 59, e14052 (2022).35398913 10.1111/psyp.14052
30. Pfurtscheller G EEG event-related desynchronization (erd) and synchronization (ers) Electroencephalogr. Clin. Neurophysiol. 1997 1 26 10.1016/S0013-4694(97)88021-6
Pfurtscheller, G. EEG event-related desynchronization (erd) and synchronization (ers). Electroencephalogr. Clin. Neurophysiol. 1, 26 (1997).10.1016/S0013-4694(97)88021-6
31. Neuper C Wörtz M Pfurtscheller G Erd/ers patterns reflecting sensorimotor activation and deactivation Prog. Brain Res. 2006 159 211 222 10.1016/S0079-6123(06)59014-4 17071233
Neuper, C., Wörtz, M. & Pfurtscheller, G. Erd/ers patterns reflecting sensorimotor activation and deactivation. Prog. Brain Res. 159, 211–222 (2006).17071233 10.1016/S0079-6123(06)59014-4
32. Cohen MX Analyzing neural time series data: theory and practice 2014 MIT press
Cohen, M. X. Analyzing neural time series data: theory and practice (MIT press, 2014).
33. Efron B Tibshirani RJ An introduction to the bootstrap 1994 Chapman and Hall/CRC
Efron, B. & Tibshirani, R. J. An introduction to the bootstrap (Chapman and Hall/CRC, 1994).
34. Altaheri H Muhammad G Alsulaiman M Physics-informed attention temporal convolutional network for EEG-based motor imagery classification IEEE Trans. Industr. Inf. 2022 19 2249 2258 10.1109/TII.2022.3197419
Altaheri, H., Muhammad, G. & Alsulaiman, M. Physics-informed attention temporal convolutional network for EEG-based motor imagery classification. IEEE Trans. Industr. Inf. 19, 2249–2258 (2022).10.1109/TII.2022.3197419
35. Formaggio E Time-frequency modulation of ERD and EEG coherence in robot-assisted hand performance Brain Topogr. 2015 28 352 363 10.1007/s10548-014-0372-8 24838817
Formaggio, E. et al. Time-frequency modulation of ERD and EEG coherence in robot-assisted hand performance. Brain Topogr. 28, 352–363 (2015).24838817 10.1007/s10548-014-0372-8
36. Riascos, J. A., Villa, D. S., Maciel, A., Nedel, L. & Barone, D. What if i had a third arm? an eeg study of a supernumerary bci system. bioRxiv 817205 (2019).
37. Oikawa T Hirano D Taniguchi T Maruyama H The effects of tool holding on body schema during motor imagery: A near-infrared spectroscopy study J. Phys. Ther. Sci. 2017 29 702 706 10.1589/jpts.29.702 28533613
Oikawa, T., Hirano, D., Taniguchi, T. & Maruyama, H. The effects of tool holding on body schema during motor imagery: A near-infrared spectroscopy study. J. Phys. Ther. Sci. 29, 702–706 (2017).28533613 10.1589/jpts.29.702
38. Inoue K Activation in the ipsilateral posterior parietal cortex during tool use: A pet study Neuroimage 2001 14 1469 1475 10.1006/nimg.2001.0942 11707103
Inoue, K. et al. Activation in the ipsilateral posterior parietal cortex during tool use: A pet study. Neuroimage 14, 1469–1475 (2001).11707103 10.1006/nimg.2001.0942
39. Miller LE Somatosensory cortex efficiently processes touch located beyond the body Curr. Biol. 2019 29 4276 4283 10.1016/j.cub.2019.10.043 31813607
Miller, L. E. et al. Somatosensory cortex efficiently processes touch located beyond the body. Curr. Biol. 29, 4276–4283 (2019).31813607 10.1016/j.cub.2019.10.043
40. Fabio C Salemme R Koun E Farnè A Miller LE Alpha oscillations are involved in localizing touch on handheld tools J. Cogn. Neurosci. 2022 34 675 686 10.1162/jocn_a_01820 35061032
Fabio, C., Salemme, R., Koun, E., Farnè, A. & Miller, L. E. Alpha oscillations are involved in localizing touch on handheld tools. J. Cogn. Neurosci. 34, 675–686 (2022).35061032 10.1162/jocn_a_01820
41. Lakshminarayanan K The effect of combining action observation in virtual reality with kinesthetic motor imagery on cortical activity Front. Neurosci. 2023 17 1201865 10.3389/fnins.2023.1201865 37383098
Lakshminarayanan, K. et al. The effect of combining action observation in virtual reality with kinesthetic motor imagery on cortical activity. Front. Neurosci. 17, 1201865 (2023).37383098 10.3389/fnins.2023.1201865
42. Alimardani M Nishio S Ishiguro H Brain-computer interface and motor imagery training: The role of visual feedback and embodiment Evolv. BCI Ther. -Engag. Brain State. Dyn. 2018 2 64
Alimardani, M., Nishio, S. & Ishiguro, H. Brain-computer interface and motor imagery training: The role of visual feedback and embodiment. Evolv. BCI Ther. -Engag. Brain State. Dyn. 2, 64 (2018).
43. Škola F Liarokapis F Embodied VR environment facilitates motor imagery brain-computer interface training Comput. Gr. 2018 75 59 71 10.1016/j.cag.2018.05.024
Škola, F. & Liarokapis, F. Embodied VR environment facilitates motor imagery brain-computer interface training. Comput. Gr. 75, 59–71 (2018).10.1016/j.cag.2018.05.024
44. Park S Ha J Kim D-H Kim L Improving motor imagery-based brain-computer interface performance based on sensory stimulation training: An approach focused on poorly performing users Front. Neurosci. 2021 15 732545 10.3389/fnins.2021.732545 34803582
Park, S., Ha, J., Kim, D.-H. & Kim, L. Improving motor imagery-based brain-computer interface performance based on sensory stimulation training: An approach focused on poorly performing users. Front. Neurosci. 15, 732545 (2021).34803582 10.3389/fnins.2021.732545
45. Choi JW Kim BH Huh S Jo S Observing actions through immersive virtual reality enhances motor imagery training IEEE Trans. Neural Syst. Rehabil. Eng. 2020 28 1614 1622 10.1109/TNSRE.2020.2998123 32634098
Choi, J. W., Kim, B. H., Huh, S. & Jo, S. Observing actions through immersive virtual reality enhances motor imagery training. IEEE Trans. Neural Syst. Rehabil. Eng. 28, 1614–1622 (2020).32634098 10.1109/TNSRE.2020.2998123
46. Gonzalez-Rosa JJ Action observation and motor imagery in performance of complex movements: Evidence from EEG and kinematics analysis Behav. Brain Res. 2015 281 290 300 10.1016/j.bbr.2014.12.016 25532912
Gonzalez-Rosa, J. J. et al. Action observation and motor imagery in performance of complex movements: Evidence from EEG and kinematics analysis. Behav. Brain Res. 281, 290–300 (2015).25532912 10.1016/j.bbr.2014.12.016
47. Athanasiou A Investigating the role of alpha and beta rhythms in functional motor networks Neuroscience 2018 378 54 70 10.1016/j.neuroscience.2016.05.044 27241945
Athanasiou, A. et al. Investigating the role of alpha and beta rhythms in functional motor networks. Neuroscience 378, 54–70 (2018).27241945 10.1016/j.neuroscience.2016.05.044
48. Medendorp WP Buchholz VN Van Der Werf J Leoné FT Parietofrontal circuits in goal-oriented behaviour Eur. J. Neurosci. 2011 33 2017 2027 10.1111/j.1460-9568.2011.07701.x 21645097
Medendorp, W. P., Buchholz, V. N., Van Der Werf, J. & Leoné, F. T. Parietofrontal circuits in goal-oriented behaviour. Eur. J. Neurosci. 33, 2017–2027 (2011).21645097 10.1111/j.1460-9568.2011.07701.x
49. Hagemann D Naumann E The effects of ocular artifacts on (lateralized) broadband power in the EEG Clin. Neurophysiol. 2001 112 215 231 10.1016/S1388-2457(00)00541-1 11165523
Hagemann, D. & Naumann, E. The effects of ocular artifacts on (lateralized) broadband power in the EEG. Clin. Neurophysiol. 112, 215–231 (2001).11165523 10.1016/S1388-2457(00)00541-1
50. Wang Z A BCI based visual-haptic neurofeedback training improves cortical activations and classification performance during motor imagery J. Neural Eng. 2019 16 066012 10.1088/1741-2552/ab377d 31365911
Wang, Z. et al. A BCI based visual-haptic neurofeedback training improves cortical activations and classification performance during motor imagery. J. Neural Eng. 16, 066012 (2019).31365911 10.1088/1741-2552/ab377d
51. Grigorev NA A BCI-based vibrotactile neurofeedback training improves motor cortical excitability during motor imagery IEEE Trans. Neural Syst. Rehabil. Eng. 2021 29 1583 1592 10.1109/TNSRE.2021.3102304 34343094
Grigorev, N. A. et al. A BCI-based vibrotactile neurofeedback training improves motor cortical excitability during motor imagery. IEEE Trans. Neural Syst. Rehabil. Eng. 29, 1583–1592 (2021).34343094 10.1109/TNSRE.2021.3102304
52. Pinardi M Impact of supplementary sensory feedback on the control and embodiment in human movement augmentation Commun. Eng. 2023 2 64 10.1038/s44172-023-00111-1
Pinardi, M. et al. Impact of supplementary sensory feedback on the control and embodiment in human movement augmentation. Commun. Eng. 2, 64 (2023).10.1038/s44172-023-00111-1
