
==== Front
Nat Commun
Nat Commun
Nature Communications
2041-1723
Nature Publishing Group UK London

52383
10.1038/s41467-024-52383-6
Article
Auditory areas are recruited for naturalistic visual meaning in early deaf people
http://orcid.org/0000-0003-1715-0147
Zimmermann Maria Mzf.zimmermann@gmail.com

12
http://orcid.org/0000-0002-5234-7415
Cusack Rhodri 3
http://orcid.org/0000-0002-2907-8042
Bedny Marina 2
http://orcid.org/0000-0002-2153-7793
Szwed Marcin m.szwed@uj.edu.pl

1
1 grid.5522.0 0000 0001 2162 9631 Institute of Psychology, Jagiellonian University, Krakow, Poland
2 https://ror.org/00za53h95 grid.21107.35 0000 0001 2171 9311 Department of Psychology and Brain Sciences, Johns Hopkins University, Baltimore, USA
3 https://ror.org/02tyrky19 grid.8217.c 0000 0004 1936 9705 Trinity College Institute of Neuroscience, Trinity College Dublin, Ireland
17 9 2024
17 9 2024
2024
15 803519 4 2023
4 9 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Congenital deafness enhances responses of auditory cortices to non-auditory tasks, yet the nature of the reorganization is not well understood. Here, naturalistic stimuli are used to induce neural synchrony across early deaf and hearing individuals. Participants watch a silent animated film in an intact version and three versions with gradually distorted meaning. Differences between groups are observed in higher-order auditory cortices in all stimuli, with no statistically significant effects in the primary auditory cortex. Comparison between levels of scrambling revealed a heterogeneity of function in secondary auditory areas. Both hemispheres show greater synchrony in the deaf than in the hearing participants for the intact movie and high-level variants. However, only the right hemisphere shows an increased inter-subject synchrony in the deaf people for the low-level movie variants. An event segmentation validates these results: the dynamics of the right secondary auditory cortex in the deaf people consist of shorter-length events with more transitions than the left. Our results reveal how deaf individuals use their auditory cortex to process visual meaning.

In people who are deaf, the parts of the brain usually responsible for processing audition can take over visual tasks. Here, the authors show that the auditory cortex in early deaf individuals processes visual meaning conveyed in naturalistic stimuli.

Subject terms

Perception
Attention
issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Studies of sensory loss provide insights into mechanisms of plasticity in the human brain. Following deafness, auditory cortices become responsive in a wide range of non-auditory tasks. These include perceptual functions, such as peripheral vision1, motion perception2,3, visual motion discrimination4 and temporal and spatial sequence processing5,6. Responses have also been found for various high-level tasks, including working memory and executive control7–10.

Despite clear evidence of functional reorganization, many questions remain unanswered about the extent and nature of plasticity. A central issue is whether different parts of the auditory network are recruited for different cognitive functions in deafness. In hearing people, there is a clear and consistent organization of the temporal cortex within and across hemispheres11,12. Within the hemisphere, there is a hierarchy of processes: primary auditory cortices are involved in low-level perception, secondary process higher-level auditory stimuli, and STS is more involved in multimodal processes including meaningful narrative processing13,14. There is also specialization across hemispheres with the left hemisphere being more engaged in processing speech, and the right being more involved in processing non-verbal auditory stimuli15,16. The degree of specialization in the auditory system of people born deaf remains unclear. Do different parts of the auditory cortices take on different functions? How much of the auditory cortices show recruitment to the visual stimuli in the deaf individuals? Does this reorganization extend into primary as well as secondary auditory cortices?

Such questions about the level of processing and the anatomical extent of the repurposing have been difficult to tackle using traditional task-based fMRI studies. Each experiment typically tests a specific cognitive process and targets a specific part of the auditory cortex that responds to the function in question. For example, several studies have failed to find responses to visual stimuli in primary auditory cortices (Bola et al.,5; Cardin et al.,8), but it is not known whether this is simply because these studies happened not to have sampled the particular processes to which A1 responds in deaf.

To get a more coherent and broader picture of the extent and nature of auditory cortex repurposing, we apply a complementary approach: data-driven analysis with naturalistic meaningful stimuli, i.e., an animated movie17. The basic tenet of this approach is that a rich, continuous stimulus, such as a story or a movie, captures a wide swath of cognitive processes, from low-level sensory perception to high-level narrative understanding14,17,18. Data-driven analysis methods can then be used to gain insight into the level of processing that maximally applies to a given cortical system. One such method quantifies inter-subject synchronization for an intact animated movie and for gradually distorted variants of the same film, to assess which broad level of cognitive functions a particular area supports. Previous studies have found that low-level sensory regions (e.g., primary auditory and visual cortices) generally exhibit similar levels of synchronization for intact and disrupted versions of the stimulus, with only a small decrease in synchrony with greater distortion17. Higher-level regions, in contrast, show a steep synchronization drop when the meaning is removed by scrambling or the temporal structure of the movie is distorted17,18. In the current study, we used this approach to test whether, across areas of auditory cortices, there is variation in the drop in synchrony with distortion. If so, this would suggest that different auditory areas occupy different positions in the cognitive processing hierarchy in deaf people14,18.

In a second approach, we used Hidden Markov Models (HMM) to derive the underlying temporal structure of the neural response to the intact animated movie across cortical areas19. This approach models neural activity as a series of discrete steady-states separated by boundaries. Previous studies have found that higher-level regions (e.g., precuneus, PFC, hippocampus) lock into longer steady states corresponding to high-level processing of a meaningful narrative19. By contrast, low-level perceptual regions (early sensory areas, such as V1) show shorter steady states, even for highly meaningful and complex stimuli such as movies19–21.

HMM and inter-subject correlation analyses are complementary. Both provide insight into the level of processing within the cortical hierarchy, but HMM analysis uses a different principle and relies solely on data from the intact meaningful movie stimulus. Consistent evidence from these two approaches would provide clear insight into the level of processing across different auditory regions in deaf individuals.

Results

To measure functional reorganization in auditory cortices and determine their position in a putative processing hierarchy, we measured cortical activity with fMRI while deaf and hearing participants viewed an animated silent movie (“The Triplets of Belleville”) as well as several distorted variants of the same movie: long scrambled (12-second chunks, scrambled in temporal order), short scrambled (2 s chunks, scrambled) and visually distorted (frame-by-frame diffeomorphic warping) version that removed meaningful content (e.g., objects) but preserved the low-level visual characteristics of the movie (see Fig. 1). All scrambled versions were created using the final 10 min of the original, intact movie, and were presented in a counterbalanced order. The entire movie was shown to participants after they had viewed the scrambled versions to prevent any potential influence on their interpretation of higher-level meaning in the scrambled content due to knowledge of the movie’s plot.Fig. 1 Study design.

A Stimuli: Participants passively watched an animated silent movie (“The Triplets of Belleville”) in one intact and three distorted versions (prepared from the last 10 min of the intact movie): (1) scrambled long (dark green) (2) scrambled short (light green) (3) diffeomorphic (yellow): visually distorted version, prepared by applying diffeomorphic image transformation to the intact movie. B Design: Three modified versions of the movie were first presented in counterbalanced order, followed by the first part of the intact movie (blue) (25 min). This was followed by an anatomical scan, after which the last part of the intact movie (blue) was presented (10 min). Images used in the figure are sourced from “Triplets of Belleville” under the authorization of the director Mr Sylvain Chomet and the company Les Armateurs for the reproduction of these images for the sole purpose of illustrating this scientific article. This authorization is granted free of charge, on a non-exclusive basis. Credits: LES TRIPLETTES DE BELLEVILLE A film by Sylvain Chomet 2002 Les Armateurs/Production Champion/Vivi Film/France 3 Cinéma/ RGP France/Sylvain Chomet. As per the terms of the reuse of these images, the images may not be modified or used in other publications. Any other use or distribution by other means known or to be known is strictly forbidden.

The original version of the movie does not include any language but does include music. In our study, the soundtrack was removed to match the experience as much as possible across deaf and hearing participants. We performed inter-subject correlation analysis separately in each group, for each of the three distorted versions of the movie, and the 10 min of the intact movie. The inter-subject correlation analysis focuses on neural activity triggered by stimuli but does not consider the specific content or features of the stimulus causing synchronization. When we observe higher ISC for intact stimuli compared to scrambled ones, it suggests that a particular brain region responds to the higher-level meaning or narrative features present in the intact stimulus, which is absent in the scrambled stimulus.

Increased synchronization of the auditory cortices in deafness

The inter-subject correlation (ISC) analysis revealed the hierarchy of visual processing across the brain. In both hearing and deaf diffeomorphic stimulus led to significant inter-subject correlation (ISC) in low-level visual regions. More meaningful stimulus types led to significant ISC in the increased part of the brain, including the temporal parietal junction, the middle temporal cortex, the precuneus, and the middle frontal cortices for the most meaningful stimulus (Fig. 2).Fig. 2 Whole brain intersubject correlation analysis.

Whole brain Inter-subject correlation maps, shown separately for each stimulus type and each group (deaf and hearing). Results show the hierarchy of cognitive processing in both groups: early visual cortices were synchronized to a similar degree across stimulus types, while higher-cognitive areas were more synchronized by the intact version. The significance was calculated using nonparametric permutation tests. The maps represent ISC (z-Fisher) significant at the level p < 0.05 FDR voxel-wise corrected, cluster size > 30 voxels.

In the deaf participants, parts of the superior temporal cortices (STC) were significantly synchronized for each stimulus type. In the hearing, significant voxels within the superior temporal gyrus were only found for the meaningful stimulus (intact movie) mostly in the right hemisphere.

We found increased synchrony for the animated film in the deaf group in a range of higher-order auditory areas, in bilateral superior and middle temporal cortices (Fig. 2; for all between-group contrasts). No significant effect was revealed in the primary auditory cortex at the whole brain level (p < 0.05, FDR corrected).

In the right hemisphere, all stimulus types, including low-level diffeomorphic stimulus, show increased ISC in the deaf compared to hearing individuals. However, in the left hemisphere, the effect was present only for the more meaningful stimuli (Fig. 2). We did not identify regions with a significant increase in ISC in hearing relative to the deaf group.

Comparisons between levels of scrambling revealed heterogeneity of function in different secondary auditory areas within and between hemispheres in the deaf group. For all voxels that showed increased synchrony in the deaf relative to the hearing group for any stimulus type (Fig. 3), we calculated a Temporal Receptive Window (TRW) Index (see Lerner et al. 14, Blank et al.22 for a similar analysis). The TRW index estimates the slope of synchrony decrease from the intact, most cognitively rich stimulus to the scrambled and diffeomorphic, and least cognitively rich stimulus (calculated as linear contrast: z_ISC intact *3 + z_ISC scrambled long - z_ISC scrambled short – z_ISC diffeomorphic*3).Fig. 3 Clusters with stronger inter-subject synchronization in the deaf relative to the hearing (deaf> hearing, one tailed permutation test).

Stronger synchronization is seen in the secondary auditory cortex (STG) of the deaf for each type of stimuli. The effect is bilateral for higher-level stimuli (intact, scrambled long) and right-lateralized for lower-level stimuli (scrambled short, diffeomorphic) Intergroup contrast was calculated using a permutation test. The maps represent significant voxels at the level p < 0.05 corrected for multiple comparisons using FDR, cluster size > 30 voxels.

Low-level visual information is present in each of the levels of scrambling. Thus, in areas sensitive to low-level visual information, ISC should be similar for each stimulus type and the TRW index should be low. The high-level regions show relatively higher ISC for the meaningful intact stimulus; in these areas, the TRW index is higher. For example, the TRW index in the primary visual areas (V1) is equal to 0.49, while the index in the higher-level area (precuneus) is equal to 2.99 (averaged across groups).

The left secondary auditory cortices showed a steeper slope than the right, suggesting a higher order of processing in the left hemisphere in the deaf individuals (Fig. 4A). For the voxels that showed increased synchrony in the deaf participants within the higher auditory cortex (Te2, Te3) the TRW index was significantly lower in the right hemisphere compared to the left hemisphere (right-left difference= .65, p = .001, two-tailed permutation test, n = 1000). This suggests that the left and right higher auditory cortices may operate at different levels of processing. The hemispheric difference was not statistically significant for voxels localized in the mid-STS (right-left difference = −.04, p = .685, two-tailed permutation test, n = 1000).Fig. 4 Temporal receptive windows analysis.

A Temporal Windows Index in the deaf group calculated as: z_ISC intact *3 + z_ISC scrambled long - z_ISC scrambled short– z_ISC diffeomorphic*3. For each voxel the TRW index was divided by the highest zISC score in this voxel. The map shows the TRW indices only in the regions of the brain which exhibit significant between group effect (ISC deaf > ISC hearing) for any stimulus type. Voxels were colored with temporal windows index: yellow represents short temporal windows, blue - long temporal windows. The auditory cortex of the left hemisphere in the deaf individuals shows higher temporal windows indices and the right hemisphere shows both lower and higher temporal windows indices. B ISC results (z-Fisher) for auditory ROIs (primary, higher auditory cortex, and mid-STS) in the deaf and hearing group separately. Higher auditory cortex ROI comprise both TE2 and TE3 regions combined. Source data are provided as a Source Data file.

The TRW index also varied within hemispheres along the medial-to-lateral and posterior-to-anterior axes. In both hemispheres, the mid-STS, a higher-order auditory region in the hearing, showed longer temporal receptive windows than earlier auditory areas. This pattern was most pronounced in the right hemisphere. (ROIs difference left = 0.61, p = 0.001, ROIs difference right = 1.20, p < 0.001, two-tailed permutation tests, n = 1000).

The region of interest (ROI) analysis comparing ISC responses across hemispheres, auditory areas (primary, higher, and mid-STS), groups, and levels of scrambling also revealed heterogeneity of function within auditory areas in the deaf group (Fig. 4B).

Nonparametric repeated measures analysis of variance with ISC as a dependent variable and four factors: ROI, hemisphere, group, and level of scrambling (stimulus type) showed the main effect of group (F(1, 41) = 43.5, p < .001, R2 = 0.04). The effect of the group interacted with the effect of hemisphere and stimulus type (interaction: group x hemisphere x stimulus type (F(2, 78) = 2.77, p = .048, R2 = 0.003) and with ROI and stimulus type (interaction: group x ROI x stimulus type (F(6, 246) = 4.61, p = 0.004, R2 = 0.008).

Post hoc pairwise permutation tests show a significant between-group difference for the low-level stimuli (short segments) in the right higher temporal cortex (deaf- hearing difference= 0.029, p = .007, 95% CI= [0.008, 0.05]), but not in the left higher temporal cortex (deaf- hearing difference=0.006, p = .487, 95% CI= [−0.013, 0.026]). More meaningful stimulus types (long segments and intact movie) show the between-group effect bilaterally (deaf- hearing difference intact left = 0.025, p < .001, 95% CI= [0.04, 0.11], deaf- hearing difference intact right = p < 0.001, 95% CI= [0.04, 0.10]), suggesting again that the left and right auditory cortex in the deaf may reorganize to process visual stimuli on different levels. Consistent with the whole-brain analyzes, an ROI analysis looking specifically at the primary auditory cortex found no significant between-group differences in this region for any stimulus type (p > .05 for any stimulus type) (Fig. 4B). In sum, a comparison of synchrony across different levels of scrambling and distortion revealed functional differentiation across and within hemispheres in the deaf group.

Despite the between-group differences, the synchronization of the auditory ROIs in the hearing shows similar patterns of response as in the deaf participants. However, in the hearing, the ISC was not significant in any stimulus types other than the intact movie, which evoked significant synchronization in mid-STS (intact right: r(21) = 0.2, 95% CI= [0.11, 0.29], p < .001, intact left: r(21) = 0.16, 95% CI= [0.05, 0.19]) p < 0.001, permutation tests, n = 1000) (Fig. 4B).

Data-driven event segmentation analysis

To validate our results, we performed an event segmentation analysis on partly independent data, a longer movie stimulus (25 min) using the Hidden Markov Model (HMM)19,23–25 This analysis uses HMM to detect neural event boundaries and characterize the temporal structure of neuronal dynamics occurring while subjects were watching the intact movie (Fig. 5A, B, see “methods”).Fig. 5 Hidden Markov Models (HMM) analysis derives the underlying temporal structure of neural responses to the intact movie.

A Different brain regions process events at a range of different time scales. Higher-level regions, for example, lock into longer steady states corresponding to high-level processing of a meaningful narrative (e.g. “gangsters kidnap the main character”). B Given a set of time courses from a region of interest, the event segmentation model temporally divides the data into “events” with stable activity patterns, punctuated by “event boundaries” (vertical dotted lines) at which activity patterns rapidly transition to a new stable pattern. C Depiction of auditory ROIs used in the HMM analysis. D Average event length in the auditory cortex ROIs n the deaf and the hearing. In the deaf the models were fitted to the auditory ROIs bilaterally (mid-STS, TE3, TE2 and TE1= primary auditory cortex) Colored ROIs represent regions of interests that show significant model fits. The empty ROI (white borders) represent regions that do not exhibit significant model fit at the level of p < 0.05, FDR corrected. The color of the colored regions represents the average segment length for the preferred event segmentation model. yellow- green represents shorter, and blue- longer temporal segments revealed in the model. The illustration was prepared using CorelDraw by an illustrator hired by the Jagiellonian University. Commercial use of the illustration under a CC-BY law is with the consent of the Jagiellonian University and the illustrator.

In both groups, we determined the preferred timescale of each region in the auditory cortex (primary auditory cortex, Te2, Te3, and mid-STS, Fig. 5C) by running HMMs with different numbers of event states and finding the average event length that yielded the best model fits across group subject. The model fit was computed as the average correlation for pairs of time points falling within the same (HMM-derived) event, minus the average correlation for pairs of time points falling in different events23. After estimating the preferred timescale (segment length) for each ROI we estimated how well the preferred model fits the data by comparing it with the permuted versions of the model boundaries (1000 permutations).

Depending on whether a particular region is involved in processing the stimulus, some regions may exhibit highly stable patterns within events across subjects. In contrast, in regions of the brain that are not particularly involved in processing naturalistic stimulus, event segmentation models do not reliably fit the data. In line with the results of the ISC analysis, in the hearing the models do not fit the data at the statistically significant level (p < 0.05) for most tested ROIs, except for the right mid-STS (Fig. 5D).

Consistent with the idea that deafness leads to the recruitment of auditory cortices for processing naturalistic visual stimuli, in the deaf, the event segmentation models were significantly well-fitted bilaterally for the area of the superior temporal sulcus (mid-STS, p < 0.01) and in the two regions of the higher auditory cortex (Te2 and Te3, p < 0.01) (Fig. 5D). The fit was not significant in the primary auditory cortex in either group.

The HMM analysis focused on the deaf group showed the heterogeneity of the level of processing across auditory areas. The left secondary auditory areas of the deaf participants show a preference for longer (fewer) events with an average segment length of 48 s in Te3 and 36 s in Te2 (30 and 40 events, respectively). In the right secondary auditory cortex, the events were shorter and more numerous: the average segment length is 22 s in Te3 and 28 s Te2 (70, and 50 events, respectively).

Both right and left higher auditory regions (mid-STS) showed longer, less numerous segments compared to the more superior secondary auditory areas in the STG (Te3, Te2). An average segment length in the deaf participants was 48 s in the right mid-STS, and 55 s in the left mid-STS. The right mid-STS showed similarly long segments in the hearing (25 segments, on average 55 s long) (see Fig. 5D).

Discussion

We found that secondary auditory areas of early deaf people respond to high-level visual information in a naturalistic animated movie. When watching a silent movie, secondary but not primary auditory cortices synchronized significantly more in deaf than hearing individuals and became less synchronized as the meaning of the stimulus was distorted by scrambling. Data-driven event segmentation using a Hidden Markov Model revealed a coherent event structure in secondary ‘auditory’ cortices of deaf people at slow and intermediate timescales. These two lines of evidence suggest that deaf people use their secondary auditory cortices to extract visual content from rich non-verbal stimuli. These findings from naturalistic stimuli, complement previous task-based studies and provide an organizing principle for several studies of task-related activations previously observed across the auditory cortex.

The responses to the naturalistic film and its distorted variants revealed functional segregation across different auditory areas in deafness. As the meaning of the movie was progressively disrupted by temporal and spatial scrambling, the left hemisphere showed a sharper fall-off in synchrony, relative to the right. Data-driven HMM analysis likewise showed longer processing time- scales in the left hemisphere, with slower event transitions. This result suggests a higher-order level of processing in left hemisphere auditory areas of deaf people since these respond maximally to high-level content at slower timescales and slower transitions in neural states.

Heterogeneity was also observed within the hemispheres. HMM revealed slower event transitions (~40 s events) in more lateral STS regions bilaterally, while the secondary auditory areas of the right hemisphere (Te2, Te3) unfolded at a faster timescale (~20 s events) and showed a less steep decrease in synchrony with scrambling. Overall, we found clear evidence of functional segregation in auditory areas, with different auditory regions engaged at different levels in the cognitive hierarchy and recruited in different ways for process visual meaning in deafness.

No evidence for primary auditory cortex engagement in visual processing

The rich and multifaceted naturalistic stimulus used in our study did not evoke a statistically significant response in the primary auditory areas in either the hearing or the deaf group at the whole-brain level. There was no statistically significant increased response in this region in the deaf group. The results do not allow for any conclusion regarding the involvement of A1 in the broad range of different visual semantic and perceptual functions engaged in the processing of an animated movie. The absence of statistically significant evidence for AI recruitment contrasts with the plasticity observed in congenital blindness, where the primary visual cortex shows robust activations for a range of cognitive tasks, including naturalistic auditory movies and stories. The lack of a statistically significant difference between groups in this region is notable in the context of the ongoing debate regarding the reorganization of primary auditory regions in deafness, though it remains consistent with multiple possible interpretations.

One possibility is that the repurposing of the auditory cortex does extend to primary auditory regions, but the relevant functions were not captured by analyzing brain activity in response to visual movies. The few studies with deaf participants that did show activations in the primary auditory cortex of deaf individuals involved fast executive processing, such as task-switching9 and double flash detection26. In the current study, the movie was viewed passively. Karns et al.26 also observed that the primary auditory area in people born deaf is more responsive to somatosensory than visual stimulation. Thus, it is also possible that A1 is recruited by somatosensation, also not captured in the current study. This hypothesis is supported by animal studies showing the recruitment of A1 for somatosensory sensing in deaf cats27,28, as well as studies on plasticity in the cochlear nucleus, which shows an increase in the number of somatosensory projections after auditory deprivation29,30.

Another possibility is that A1 in deaf individuals does not get reorganized and remains functionally dormant. In this scenario, the cross-modal plasticity in the deaf stops at the boundaries of the secondary auditory cortex in the posterior part of the superior temporal gyrus (STG)31. In fact, activations of the primary auditory cortex in the deaf induced by visual or tactile tasks are generally modest or absent. Various tasks, including language32 visual semantic task10, working memory8 face recognition33 visual motion perception4 and tactile and visual sensory discrimination5,6 fail to evoke activation in A1. Analogously, the primary auditory cortex in the deaf cat also does not show the effects of cross-modal reorganization34,35. It is thus possible that primary auditory areas in deaf people do not assume any new non-auditory functions. Indeed, animal studies report an atrophy of the deep layers of the auditory cortex of deaf cats which disrupts long-range connections, restricting communication with higher-order auditory areas35,36. If human atrophy of this nature is indeed present, it could preclude repurposing. Anatomical post-mortem and high-field quantitative MRI are needed to verify this hypothesis.

Importantly, in most studies that fail to uncover significant effects in the primary auditory cortex among deaf individuals, the primary focus is not on A1. In these studies, the region of interest in the Heschl’s gyrus is typically determined at the group level using anatomical atlases4,5. In contrast, in two studies that did reveal significant effects, the regions of interest were defined at the individual level9,26. One possible interpretation is that the effects in the primary auditory cortex may go undetected at the group level due to the anatomical variability in this particular region.

Heterogeneity across hemispheres

We found clear evidence of differences in functional specialization of high-level auditory areas in deaf people. The current finding of heterogeneity between auditory areas of the hemispheres is consistent with previous studies showing that higher-level tasks, such as sign language and working memory, can evoke greater activation in the left superior temporal cortex (STC) in deaf participants8,32,37,38 while visual motion, spatial and lower-level processing evoke greater activation in the right STC39,40. Interestingly, studies with animals find no such hemispheric specialization. In deaf cats, left and right secondary and primary auditory cortices do not differ in response to visual41 or somatosensory stimulation28. This suggests that the lateralization in the temporal cortex in deaf individuals may be unique for humans and could be related to the evolutionary predisposition for language and speech in the left hemisphere.

In hearing, the mid-STS of the left hemisphere is maximally responsive to spoken language and unresponsive to a wide range of non-linguistic tasks (e.g., visual working memory, social and numerical reasoning)42–44. However, recent evidence suggests that even in the hearing, non-verbal meaningful events depicted in movies45 and pictures46 do engage language systems in the STS and elsewhere, albeit less than language stimuli. Language regions might therefore be poised to process meaningful events, whether conveyed by language or through images, and deafness enhances those responses to visual meaning. The current findings suggest that in deafness, responses to visual meaning also expand superiorly into secondary auditory areas, which are thought to process lower-level aspects of speech in the hearing47 Enhancement and expansion of responses to visual narratives in deafness could be related to deafness per se, to different modalities of language use across hearing and deaf populations, or both. Since linguistic processing in deaf signers is based on visuo-spatial modality involving human motion and facial expression, language areas in the temporal cortex may enhance capacities to extract meaning from nonverbal visual narratives. Indeed, the “Triplets of Belleville” plot is rich in various meaningful visual cues, social behavior, body movement, and gestures. In line with this interpretation, previous studies showed that deaf signers show higher responsiveness to non-verbal meaning (gesture) in the left lateralized auditory areas including the temporal cortex48. In sum, higher-order processing of visual meaning in left than right lateral temporal cortices of early deaf people may be related to intrinsic predispositions of these cortical areas for language in humans.

Another possibility is that the right secondary auditory regions might be taken over by lower-level visuo-spatial attention mechanisms. These mechanisms are known to be right lateralized49. Task-based studies on deafness show significant right posterior STG activations for several attention-related functions and perceptual tasks such as spatial and temporal sequence discrimination5,6, visual motion detection4, visuospatial working memory50.

Visually-driven attention is also right-lateralized in the hearing and is usually associated with regions proximal to superior temporal cortex, mostly the parietal junction (TPJ)49,51,52. It has been previously proposed that the proximity of TPJ to the superior temporal cortex may explain the activation for visual attention in the right STC in deaf participants9,53. Consistent with this proposal, in our data, these right-hemisphere parietal areas, in the deaf as well as in the hearing, show synchrony for the short-movie fragments (Fig. 2). This indicates that our short-fragment stimuli engage the type of attentional mechanisms in question. The proximity of TPJ and temporal auditory areas could then provide the basis for the recruitment of the right STG (Te2, Te3) for spatial attention-related functions in deaf individuals. In this scenario, the attentional functions based on the right TPJ would ‘invade’ its cortical neighbors.

Heterogeneity within hemisphere

Our data also shows a hierarchy of processing within the auditory network of early deaf people. Higher-level regions in the lateral STS showed a steeper fall-off in synchrony and longer processing time scales in data-driven HMM analysis. The secondary auditory areas, in contrast, showed a shallower fall-off slope and shorter processing time scales. The difference between secondary auditory areas and more lateral STS regions was most pronounced in the right hemisphere but also present in the left.

This divergence of functions between different regions of the higher auditory cortex in deaf individuals is in part consistent with the cortical organization in the hearing. There is ample evidence that regions in the STS are at least in part multimodal and engaged in processing non-verbal visual meaning in the hearing population. The right STS responds to non-verbal components of a narrative: face-voice matching54, gesture and face movement55, and meaningful biological motion56,57. This was also observed in the current data, where the right mid-STS shows significant synchronization in both deaf and hearing, and the HMM event segmentation revealed the coherent event structure in this region in both groups. The structure of events in the right posterior STS in deaf and hearing shows high consistency in the times when events occur. In both groups, the right STS is likely to be engaged in processing higher-level visual meaning, with stronger engagement in deaf people.

Partly consistent with this view, a recent study using naturalistic stimuli (movies) show a similar pattern of activation in a group of deaf, blind, and hearing- sighted individuals independently on the modality in which they perceived a movie58.

Our results suggest that the engagement of STS in processing the meaning may be enhanced in the deaf individuals and expand to the lower-level superior temporal cortex. The more unimodal part of the superior temporal cortex, including STG (Te3, Te2) exhibits coherent event structure only in the deaf and not in the hearing and shows an increased synchronization in the deaf compared to hearing participants.

This heterogeneity across different parts of the reorganized auditory cortex contrasts with the findings in the blind. An analogous naturalistic stimuli study on the role of the visual cortex in blind individuals showed that their primary visual cortex is synchronized exclusively for intact auditory movies. Unlike the current study on deaf people, this study also found no evidence of processing hierarchy differences across the visual cortices of people born blind. In the blind, only the meaningful, intact movie led to significant synchronization of the visual cortices. These results suggest important differences in cross-modal reorganization across sensory systems.

In the present study, using a rich naturalistic stimulus, we were able to capture the nature of auditory cortex repurposing across a range of different time scales and levels of meaning. Notably, both our analytical approaches, ISC and HMM, consistently showed the engagement of the auditory cortex in processing visual meaning in deaf individuals. The right hemisphere synchronized for both high- and low-level stimuli, while the left hemisphere specialized in higher-level narrative processing. In the right hemisphere, we found the secondary auditory regions involved in lower-level processing and shorter temporal windows, and right STS being engaged in higher-level processing. Overall, the heterogeneity of the auditory cortex is at least partly consistent with the specialization of the temporal cortex in the hearing. These data suggest that in the absence of audition, secondary auditory areas become engaged in extracting meaning from visually presented events and subspecialize for different aspects of meaning extractions across and within the hemisphere.

Methods

The study was conducted in compliance with ethical regulations, the study protocol has been approved by the Committee of Research Ethics, Jagiellonian University, Krakow, Poland.

Participants

21 early deaf participants (mean age = 31.7, SD = 5.4, 10 women) and 22 hearing participants (mean age = 29.6, SD = 5.1, 10 women) took part in the study. The sex of participants was determined based on self-report, participants were free to not provide this information. The data were collected to ensure similar sex/gender distribution across deaf and hearing groups. No further analysis of sex or gender was carried out.

Inclusion criteria for deaf participants include prelingual deafness (onset of deafness < 8 months), severe level of deafness (above 90 dB threshold in both ears), and lack of a cochlear implant. All participant data were analyzed in the stimuli-driven analysis (inter-subject correlation). One deaf participant’s data was removed from the second data-driven analysis (HMM analysis), due to extensive head movement and missing data (the participant did not finish the last intact movie run). The control group consisted of 22 non-signers of hearing that matched the early deaf signer group in age, level of education, and sex. All deaf participants reported acquiring Polish Sign Language (PJM) as their first language. Ten participants reported having deaf parents and acquired PJM (Polish Sign Language) from them as the first language, while 11 participants acquired PJM in early childhood (3–6 years old) as the first language.

For detailed information on deafness etiology and language experience, see Tables (Tables 1 and 2). Instructions were given orally for hearing and in Polish Sign Language (PJM) for deaf participants (by use of a sign language interpreter). Written informed consent was signed by all participants. Participants received a compensation of 200PLN for the study participation.Table. 1 Group of early deaf individuals: demographics

	sex	age	education	level of deafness	onset of deafness	cachlear implant	hearing aids	how well do you hear speech using hearing aids	
1	F	31–35	higher	> 120 dB	congenital	No	No		
2	M	36–40	higher	90–119 dB	congenital	No	No		
3	F	36–40	higher	90–119 dB	congenital	No	Sometimes	poor	
4	M	31–35	higher	90–119 dB	congenital	No	Yes	poor	
5	M	26–30	vocational	90–19 dB	5 months	No	No		
6	M	31–35	vocational	> 120 dB	congenital	No	No		
7	F	36–40	secondary	90–119 dB	congenital	No	No		
8	M	21–25	secondary	> 120 dB	congenital	No	No		
9	M	26–30	higher	90–119 dB	congenital	No	Sometimes	poor	
10	M	31–35	higher	90–119 dB	congenital	No	No		
11	M	26–30	higher	> 120 dB	congenital	No	Sometimes	poor	
12	M	31–35	vocational	> 120 dB	congenital	No	No		
13	F	36–40	higher	90–119 dB	congenital	No	No		
14	F	25–30	higher	> 120 dB	congenital	No	No		
15	M	31–35	secondary	> 120 dB	8 months	No	No		
16	M	31–35	higher	90–119 dB	6 months	No	Yes	poor	
17	M	36–40	secondary	90–119 dB	congenital	No	Yes	moderate	
18	F	21–25	higher	> 120 dB	congenital	No	No		
19	F	26–30	vocational	90–119 dB	8 months	No	Yes	poor	
20	F	21–25	higher	90–119 dB	congenital	No	Yes	poor	
21	F	31–35	higher	90–119 dB	congenital	No	No		

Table. 2 Group of early deaf individuals: language experience

	deaf mother	deaf father	PJM native	how well do you understand polish speech?	how well do you speak polish	Polish Sign Language fluency	first language exposure	first language	
1	No	No	No	poorly	poorly	fluent	3–6 yo	PJM	
2	No	No	Yes	moderate	poorly	fluent	3–6 yo	PJM	
3	Yes	Yes	Yes	poorly	poorly	fluent		PJM	
4	No	No	No	well	moderate	fluent	3–6 yo	PJM	
5	No	No	No	poorly	poorly	fluent	3–6 yo	PJM	
6	Yes	Yes	Yes	poorly	poorly	well		PJM	
7	No	Yes	Yes	poorly	poorly	fluent		PJM	
8	Yes	Yes	Yes	poorly	poorly	fluent		PJM	
9	Yes	Yes	Yes	poorly	poorly	well		PJM	
10	Yes	Yes	Yes	moderate	poorly	fluent		PJM	
11	Yes	Yes	Yes	poorly	well	fluent		PJM	
12	Yes	Yes	Yes	poorly	poorly	fluent		PJM	
13	Yes	Yes	Yes	moderate	poorly	fluent		PJM	
14	Yes	Yes	Yes	poorly	poorly	fluent		PJM	
15	No	No	No	poorly	poorly	fluent	3–6 yo	PJM	
16	No	No	No	moderate	poorly	fluent	3–6 yo	PJM	
17	No	No	No	moderate	well	well	3–6 yo	PJM	
18	Yes	Yes	Yes	poorly	poorly	fluent		PJM	
19	No	No	No	moderate	moderate	fluent	3–6 yo	PJM	
20	Yes	Yes	Yes	moderate	well	fluent		PJM	
21	Yes	Yes	Yes	poorly	poorly	fluent		PJM	

StimuliIntact stimulus was generated by shortening the Triplets of Belleville to 35 min by removing scenes that were not directly relevant to the plot line.

A piecewise scrambled version with long segments (12 s, +/− 2 s): The movie was cut into chunks, and the segments were then shuffled and displayed in pseudo-random order.

A piecewise scrambled version with short segments 2 s (+/− 1 s): similar to above, but with shorter segments.

A diffeomorphic scrambled condition. A stimulus lacking any meaning was generated by distorting the video using the diffeomorphic warping method59. Unlike conventional methods for generating low-level control stimuli (phase scrambling or texture scrambling), diffeomorphic warping preserves the basic perceptual properties of the image while removing meaning. The diffeomorphic stimulus is perceptually similar to the intact movie in terms of low-level visual feature regions (e.g., spatial frequency, spatial organization) and should yield a similar response in early visual cortices.

The modified versions were presented first in counterbalanced order. The intact movie was displayed in two parts (25’) and (10’) always following the scrambled and diffeomorphic conditions to ensure that participants cannot derive any higher-level meaning from distorted stimuli (Fig. 1).

Data Acquisition

MRI structural and functional data of the whole brain were collected on a 3 Tesla Siemens MAGNETOM Tim Trio scanner. The T1-weighted structural images were collected in 176 axial slices with 1 mm isotropic voxels using a magnetization -prepared rapid gradient echo (MP RAGE). Functional images were collected using a gradient echo planar imaging (EPI) sequence (36 sequential ascending axial slices, repetition time (TR) 1.4 s, echo time (TE) 30 ms, flip angle 70°, field of view (FOV), matrix 76 × 70, voxel size 2.5 × 2.5 × 2.5 mm, PE direction L/R, multiband MB = 4. Data analyzes were performed using fmriprep (freesurfer), python (v. 3.7), the Brain Imaging Analysis Kit, http://brainiak.org60,61, Human Brain project siibra https://siibra-python.readthedocs.io, data analysis toolboxes (nilearn, nltools) and Rstudio (R 4.0.4).

Data Analysis

Preprocessing

We performed minimal preprocessing using fmriprep62

Anatomical data preprocessing

The T1-weighted (T1w) image was corrected for intensity non-uniformity (INU)with ‘N4BiasFieldCorrection‘ and used as T1w-reference throughout the workflow. The T1w-reference was then skull-stripped with a Nipype implementation of the ‘antsBrainExtraction.sh‘ workflow, using OASIS30ANTs as target template.

Brain tissue segmentation was performed on the brain extracted T1w using ‘fast‘ [FSL 5.0.9]. Volume-based spatial normalization to one standard space (ICBM 152 Nonlinear Asymmetrical template version 2009c) was performed through nonlinear registration with ‘ants Registration‘ (ANTs 2.3.3). The BOLD reference was co-registered to the T1w reference using ‘flirt‘ [FSL 5.0.9]. Co-registration was configured with nine degrees of freedom to account for distortions remaining in the BOLD reference. The BOLD time-series were resampled onto their original, native space by applying the transforms to correct for head motion. The BOLD time-series were resampled into standard space, generating a preprocessed BOLD run in MNI152NLin2009cAsym space.

Functional data preprocessing

First, a reference volume and its skull-stripped version were generated using a custom methodology of fMRIPrep. Several confounding time series were calculated based on the preprocessed BOLD: framewise displacement (FD), DVARS, and three region-wise global signals. FD was computed using two formulations following Power (absolute sum of relative motions) and Jenkinson (relative root mean square displacement between affines). FD and DVARS are calculated for each functional run, both using their implementations in Nipype. The three global signals are extracted within the CSF, the WM, and the whole-brain masks. Additionally, a set of physiological regressors was extracted to allow for component-based noise correction.

Principal components are estimated after high-pass filtering the preprocessed BOLD time-series (using a discrete cosine filter with 128 s cut-off) for the two *CompCor* variants: temporal (tCompCor) and anatomical (aCompCor). tCompCor components are then calculated from the top 2% variable voxels within the brain mask. For aCompCor, three probabilistic masks (CSF, WM and combined CSF + WM) are generated in anatomical space.

Finally, these masks are resampled into BOLD space and binarized by thresholding at 0.99 (as in the original implementation). The confound time series derived from head motion estimates and global signals were expanded with the inclusion of temporal derivatives and quadratic terms for each. Frames that exceeded a threshold of 0.5 mm FD or 1.5 standardized DVARS were annotated as motion outliers.

Denoising

After preprocessing, we smoothed the data (fwhm=6 mm) and performed voxel-wise denoising using a GLM. To perform denoising, we fit a voxel-wise general linear model (GLM) for each participant. The effects of motion estimated during the realignment step using an expanded set of 24 motion parameters (six demeaned realignment parameters, their squares, their derivatives, and their squared derivatives), We also include dummy codes for spikes identified from global signal outliers and outliers identified from frame differencing (i.e., temporal derivative). As a common practice in naturalistic fMRI data analysis, we did not perform high-pass filtering. Instead we included linear & quadratic trends, and mean activity from a cerebral spinal fluid mask, activity to remove additional physiological and scanner artifacts63.

Whole-brain Inter-subject Correlation (ISC) Analysis

The aim of this analysis is to evaluate the degree of stimulus-driven synchronization (correlation) to that same voxel in other people’s cortices. In the whole brain level analysis for each voxel, the inter-subject correlation was calculated using the leave-one-out method. First, voxelwise synchrony was calculated as the average Pearson correlation coefficient (r) between the time course of one subject and the average time course of the rest of the experimental group (Hasson et al.,17; Lerner et al.,14). This procedure was repeated for all subjects in a group for each condition separately (intact, piecewise scrambled long, short, diffeomorphic) Results were averaged across participants. The average ISC maps with values r were transformed to Fisher’s z values to allow comparisons of correlations between groups and conditions. Differences in synchronization between stimuli and between groups were compared by subtracting the relevant z-maps. Because ISC analysis violates the assumptions required for parametric methods, we perform non-parametric hypothesis ISC testing using permutation tests14,17,64,65. Within group ISC maps (Fig. 2) were created using bootstrapping (implemented in BriainIAK, isc_bootstrap) For the between group comparisons a null distribution was created by permuting the original data using the permutation tests (implemented in BriainIAK: isc_permutation) (Fig. 3). All ISC whole-brain maps were thresholded at the level p < 0.05, FDR corrected voxel-wise, cluster size > 30 voxels.

To investigate where different parts of the auditory cortex are located in the processing hierarchy in the deaf population, we calculated the temporal receptive window index (TRW index) for voxel which show significant between group effect for any stimuli type (Fig. 4). To this end, we calculated differences in synchronization between more meaningful and less meaningful stimulus types by calculating a linear contrast across conditions ordered by degree of meaning. An analogous analysis was previously performed by calculating the difference between synchronization for high and low level stimulus14,22. Here we additionally took into consideration intermediate stimuli level, by subtracting all respective z_ISC scores (e.g. z_ISC for the intact movie compared to z_ISC for scrambled long stimulus, z_ISC for scrambled long compared to zISC diffeomorphic, etc.) Adding all these differences together resulted in the following. TRW index = 3* z_ISC intact + z_ISC scrambled_long- z_ISC scrambled_short – 3* z_ISC diffeomorphic. Note that this linear contrast captures the first moment of the differences and would be insensitive to higher order moments. To control for differences in ISC between different parts of the brain the TRW indexes were normalized. For each voxel the TRW index was divided by the highest zISC score in this voxel. The differences in TRW indexes across hemispheres (between significant voxels in the higher auditory cortex and mid-STS) were calculated using permutation randomization tests (number of permutations =1000).

ROI analysis of inter-subject correlation (ISC)

To compare the level of auditory cortex synchronization between subjects for different conditions (stimulus types) and between different parts of the temporal cortex, we proceed with the ISC analysis on the temporal cortex parcellation (see ROI definition). We performed the ISC analysis on anatomical ROIs. For each participant and each ROI, a time course was obtained by averaging throughout the region. Similarly, as in the whole brain analysis, for each ROI, each participant’s ROI time course was correlated with the average ROI time course of all participants in a group (deaf and hearing separately). The results were averaged among the participants. The averaged ISC r values were transformed to Fisher’s z values. The z-ISC values for each ROI were entered into the nonparametric permutation analysis of variance (permANOVA)66 with four factors: group, ROI (3 levels: primary, early, higher auditory cortex), hemisphere, and stimulus type (4 levels). This type of analysis of variance allowed us to fit a multifactorial model to data that are not normally distributed: a null distribution for each comparison was estimated using permutations (number of permutations =10,000). Four main effects and two interaction effects were calculated (ROI x stimulus type x group and hemisphere x stimulus type x group). We then performed post hoc pairwise comparisons: we tested the difference between the groups for each of 3 ROI s and four conditions (stimulus type) separately using nonparametric permutation tests (number of permutations =1000), p-values were adjusted for multiple comparisons using FDR correction.

Event segmentation using Hidden Markov Models

Next, we proceed with a data-driven analysis of the intact movie data. For this analysis we used a larger portion of data: the first 25 min (first part, 1035 TRs) of the entire movie. We used the larger portion of data to ensure enough power and test our hypotheses on partially independent part of stimulus. The analysis assumes that when watching continuous stimuli (movies), humans automatically divide the continuous stream of perception into segments. The time scale of these segments may be derived from the brain signal for different regions of the brain. This time scale should largely match the hierarchy of inter-subject correlation coefficients revealed from the stimuli-driven approach explained above.

We performed HMM analysis on the auditory ROIs (STS1 (mid-STS), Te3, Te2, Te1) defined from Juelich. Additionally, the analysis was performed in parcels throughout the cortex67 (supplementary material).

HMM models were estimated for each of the four ROIs separately. The number of events (i.e. shifts in activation patterns) for each time series and each ROI was estimated using the Hidden Markov Model using Brainiak HMM module. We use the procedure for model fitting as explained by Baldassano et al.19 and Williams et al.23 (ROI analysis). First, the time course was obtained from each voxel of the ROI19,23 For each ROI, the event segmentation model was applied to group-averaged data from all but one subject (using event segmentation function implemented in Brainiak). We measured the robustness of the boundaries by testing whether the event segmentation explained the temporal structure in the left-out subject. We first measured the correlation between all pairs of time points (separated by the time windows shorter than the shortest time window in the model) and then sorted these correlations according to whether the pair of time points were within the same event or crossed over an event boundary. The average difference between the within-event versus across-event correlations was used to measure how well the learned boundaries captured the temporal structure of the left-out subject. The analysis was repeated for every left-out subject and a varying number of events from k = 10 to k = 90. After averaging the results across subjects, the number of events with the highest within- versus across-event correlations was chosen as the optimal number of events for this region.

For the preferred model, we calculated the significance of the fit by comparing within- versus across-event correlations for the model boundaries to within- versus across-event correlations for the permuted versions of the model boundaries (1000 permutations).

ROIs definition

The auditory cortex ROIs were defined using the new Juelich, Human Brain Project parcellation68,69. In both ROI analyzes (ISC and HMM) we used the ROIs located in the temporal cortex which are involved mostly in processing auditory stimuli in the hearing population69. These are 3 anatomically and functionally distinct structures: one located along the Heschl gyrus (Te1.0, Te1.1 and Te1.2)–primary auditory cortex, the second along the superior temporal gyrus–secondary auditory cortex (Te2 and Te3) and the last in the superior temporal sulcus (STS1, which we call mid-STS for more clarity) -see supplementary Fig. S5. In the HMM analysis, the higher auditory cortex was further divided into secondary (Te2) and higher region (Te3) to allow more fine-grain testing for a gradient.

Controlling for the effect of stimuli order

To investigate if there was an effect of “time-on-task”, we analyzed the synchronization elicited by the first part (10 min) and the last 10 min of the intact movie. We found that the first part of the intact movie led to significantly higher ISC than the last part of the movie (Fig. 2B). Led by this finding, we performed an additional control experiment to check whether the difference between the level of whole-brain inter-subject synchronization evoked by different parts of the movie comes from the effect of order or some inherent properties of different parts of the film. Nine of the 22 hearing participants took part in this additional control fMRI experiment. The first and last parts of the movie were presented in a counterbalanced order (Fig. 1, supplementary Fig. S1A). Additionally, we presented the three control conditions that were diffeomorphic and piecewise scrambled versions of the first part of the movie, in contrast to the main experiment (Fig. 1), where the control conditions used the last part of the movie.

We found that the last part of the movie evokes a significantly lower inter-subject correlation than the first part, even if controlled for the effect of order (Fig. 1). Critically, scrambled and diffeomorphic versions of the first and last part of the movie evoked similar ISC levels. Given these results, we decided to use the first part of the movie (first 10 min) as the main intact stimulus in all subsequent analyzes. The results of an analogous analysis performed on the last part of the intact movie are described in the Supplementary Material (Fig. S1B).

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Supplementary information

Supplementary Information

Peer Review File

Reporting Summary

Source data

Source Data

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-024-52383-6.

Acknowledgements

The study was supported by National Science Center -NCN grant no. 2018/30/A/HS6/00595 to MS and a grant from the Faculty of Philosophy under the Strategic Program Excellence Initiative at Jagiellonian University. We thank Sylvain Chomet, director of the movie “Les Triplettes de Belleville,” and the movie producers: © 2002 Les Armateurs/Production Champion/Vivi Film/France 3 Cinéma/RGP France/Sylvain Chomet for authorizing the use of a still from the movie in the current article. We thank Samuel Nowak, the author of the illustration in panel A of Fig. 5. We thank Christopher Baldassano for his suggestions and explanations regarding the hidden Markov model analysis, Dawid Droździel and Bartosz Kossowski for technical support, Małgorzata Bener, Marta Rodziewicz for administrative assistance, the deaf and hearing individuals who participated in this research, and the deaf community for its support of this research; without them this work would not have been possible.

Author contributions

M.S., M.B., M.Z., and R.C. conceived the original idea and planned the experiments. M.Z. prepared the scrambled and intact stimuli; R.C. set up the diffeomorphic transformation of the movie. M.Z. collected the data, analyzed the data, and produced figures. M.Z. wrote the first version of the manuscript with support from M.B. and M.S., M.B., R.C., M.S., and M.Z. revised the manuscript. MS. supervised the project and acquired funding, and M.B. helped to supervise the project.

Peer review

Peer review information

Nature Communications thanks Samuel Nastase, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Data availability

The region-of-interest labels, and intersubject correlation maps have been deposited in the Center of Open Science database: 10.17605/OSF.IO/4HN3Z. Source data are provided with this paper as a source data file. The raw fMRI data will be available under restricted access and will be released after a 12-month embargo following publication. While the data will not be publicly available during the embargo period, the data are nonetheless available immediately upon request from the corresponding author. After the embargo period, we will fully release the data in the OpenNeuro Repository. Source data are provided in this paper.

Code availability

The code generated to produce statistical results (permANOVA, permutation tests) and figures is available on the open database: 10.17605/OSF.IO/4HN3Z. Additionally, we used code that is publicly available at the Git repository: https://github.com/naturalistic-data-analysis.

Competing interests

The authors declare no competing interests.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Finney EM Fine I Dobkins KR Visual stimuli activate auditory cortex in the deaf Nat. Neurosci. 2001 4 1171 1173 10.1038/nn763 11704763
Finney, E. M., Fine, I. & Dobkins, K. R. Visual stimuli activate auditory cortex in the deaf. Nat. Neurosci. 4, 1171–1173 (2001).11704763
2. Scott, G. D., Karns, C. M., Dow, M. W., Stevens, C. & Neville, H. J. Enhanced peripheral visual processing in congenitally deaf humans is supported by multiple brain regions, including primary auditory cortex. Front Hum Neurosci. 8, 177 (2014).
3. Shiell MM Champoux F Zatorre RJ The right hemisphere planum temporale supports enhanced visual motion detection ability in deaf people: evidence from cortical thickness Neural Plasticity 2016 2016 1 9 10.1155/2016/7217630
Shiell, M. M., Champoux, F. & Zatorre, R. J. The right hemisphere planum temporale supports enhanced visual motion detection ability in deaf people: evidence from cortical thickness. Neural Plasticity 2016, 1–9 (2016).
4. Benetti S Visual motion processing recruits regions selective for auditory motion in early deaf individuals NeuroImage 2021 230 117816 10.1016/j.neuroimage.2021.117816 33524580
Benetti, S. et al. Visual motion processing recruits regions selective for auditory motion in early deaf individuals. NeuroImage 230, 117816 (2021).33524580
5. Bola Lukasz Task-specific reorganization of the auditory cortex in deaf humans Proc. Natl Acad. Sci. 2017 114 E600 E609 10.1073/pnas.1609000114 28069964
Bola, Lukasz et al. Task-specific reorganization of the auditory cortex in deaf humans. Proc. Natl Acad. Sci. 114, E600–E609 (2017).28069964
6. Zimmermann M The extent of task specificity for visual and tactile sequences in the auditory cortex of the deaf and hard of hearing J. Neurosci. 2021 41 9720 9731 10.1523/JNEUROSCI.2527-20.2021 34663627
Zimmermann, M. et al. The extent of task specificity for visual and tactile sequences in the auditory cortex of the deaf and hard of hearing. J. Neurosci. 41, 9720–9731 (2021).34663627
7. Andin, J., Holmer, E., Schönström, K. & Rudner, M. Working memory for signs with poor visual resolution: fMRI evidence of reorganization of auditory cortex in deaf signers. Cerebral Cortex 31, 3165–3176 (2021).
8. Cardin V The organization of working memory networks is shaped by early sensory experience Cereb. Cortex 2018 28 3540 3554 10.1093/cercor/bhx222 28968707
Cardin, V. et al. The organization of working memory networks is shaped by early sensory experience. Cereb. Cortex 28, 3540–3554 (2018).28968707
9. Manini B Sensory experience modulates the reorganization of auditory regions for executive processing Brain 2022 145 3698 3710 10.1093/brain/awac205 35653493
Manini, B. et al. Sensory experience modulates the reorganization of auditory regions for executive processing. Brain 145, 3698–3710 (2022).35653493
10. Twomey T Waters D Price CJ Evans S MacSweeney M How auditory experience differentially influences the function of left and right superior temporal cortices J. Neurosci. 2017 37 9564 9573 10.1523/JNEUROSCI.0846-17.2017 28821674
Twomey, T., Waters, D., Price, C. J., Evans, S. & MacSweeney, M. How auditory experience differentially influences the function of left and right superior temporal cortices. J. Neurosci. 37, 9564–9573 (2017).28821674
11. Wessinger CM Hierarchical organization of the human auditory cortex revealed by functional magnetic resonance imaging J. Cogn. Neurosci. 2001 13 1 7 10.1162/089892901564108 11224904
Wessinger, C. M. et al. Hierarchical organization of the human auditory cortex revealed by functional magnetic resonance imaging. J. Cogn. Neurosci. 13, 1–7 (2001).11224904
12. Zatorre RJ Belin P Penhune VB Structure and function of auditory cortex: music and speech Trends Cogn. Sci. 2002 6 37 46 10.1016/S1364-6613(00)01816-7 11849614
Zatorre, R. J., Belin, P. & Penhune, V. B. Structure and function of auditory cortex: music and speech. Trends Cogn. Sci. 6, 37–46 (2002).11849614
13. de Heer WA Huth AG Griffiths TL Gallant JL Theunissen FE The hierarchical cortical organization of human speech processing J. Neurosci. 2017 37 6539 6557 10.1523/JNEUROSCI.3267-16.2017 28588065
de Heer, W. A., Huth, A. G., Griffiths, T. L., Gallant, J. L. & Theunissen, F. E. The hierarchical cortical organization of human speech processing. J. Neurosci. 37, 6539–6557 (2017).28588065
14. Lerner Y Honey CJ Silbert LJ Hasson U Topographic mapping of a hierarchy of temporal receptive windows using a narrated story J. Neurosci. 2011 31 2906 2915 10.1523/JNEUROSCI.3684-10.2011 21414912
Lerner, Y., Honey, C. J., Silbert, L. J. & Hasson, U. Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. J. Neurosci. 31, 2906–2915 (2011).21414912
15. Albouy P Benjamin L Morillon B Zatorre RJ Distinct sensitivity to spectrotemporal modulation supports brain asymmetry for speech and melody Science 2020 367 1043 1047 10.1126/science.aaz3468 32108113
Albouy, P., Benjamin, L., Morillon, B. & Zatorre, R. J. Distinct sensitivity to spectrotemporal modulation supports brain asymmetry for speech and melody. Science 367, 1043–1047 (2020).32108113
16. Belin P Lateralization of speech and auditory temporal processing J. Cogn. Neurosci. 1998 10 536 540 10.1162/089892998562834 9712682
Belin, P. et al. Lateralization of speech and auditory temporal processing. J. Cogn. Neurosci. 10, 536–540 (1998).9712682
17. Hasson U Intersubject synchronization of cortical activity during natural vision Science 2004 303 1634 1640 10.1126/science.1089506 15016991
Hasson, U. Intersubject synchronization of cortical activity during natural vision. Science 303, 1634–1640 (2004).15016991
18. Hasson U Yang E Vallines I Heeger DJ Rubin N A hierarchy of temporal receptive windows in human cortex J. Neurosci. 2008 28 2539 2550 10.1523/JNEUROSCI.5487-07.2008 18322098
Hasson, U., Yang, E., Vallines, I., Heeger, D. J. & Rubin, N. A hierarchy of temporal receptive windows in human cortex. J. Neurosci. 28, 2539–2550 (2008).18322098
19. Baldassano C Discovering event structure in continuous narrative perception and memory Neuron 2017 95 709 721.e5 10.1016/j.neuron.2017.06.041 28772125
Baldassano, C. et al. Discovering event structure in continuous narrative perception and memory. Neuron 95, 709–721.e5 (2017).28772125
20. Cohen SS Tottenham N Baldassano C Developmental changes in story-evoked responses in the neocortex and hippocampus eLife 2022 11 e69430 10.7554/eLife.69430 35787304
Cohen, S. S., Tottenham, N. & Baldassano, C. Developmental changes in story-evoked responses in the neocortex and hippocampus. eLife 11, e69430 (2022).35787304
21. Lee CS Aly M Baldassano C Anticipation of temporally structured events in the brain eLife 2021 10 e64972 10.7554/eLife.64972 33884953
Lee, C. S., Aly, M. & Baldassano, C. Anticipation of temporally structured events in the brain. eLife 10, e64972 (2021).33884953
22. Blank IA Fedorenko E No evidence for differences among language regions in their temporal receptive windows NeuroImage 2020 219 116925 10.1016/j.neuroimage.2020.116925 32407994
Blank, I. A. & Fedorenko, E. No evidence for differences among language regions in their temporal receptive windows. NeuroImage 219, 116925 (2020).32407994
23. Williams JA High-order areas and auditory cortex both represent the high-level event structure of music J. Cogn. Neurosci. 2022 34 699 714 10.1162/jocn_a_01815 35015874
Williams, J. A. et al. High-order areas and auditory cortex both represent the high-level event structure of music. J. Cogn. Neurosci. 34, 699–714 (2022).35015874
24. Loiotile RE Cusack R Bedny M Naturalistic audio-movies and narrative synchronize “visual” cortices across congenitally blind but not sighted individuals J. Neurosci. 2019 39 8940 8948 10.1523/JNEUROSCI.0298-19.2019 31548238
Loiotile, R. E., Cusack, R. & Bedny, M. Naturalistic audio-movies and narrative synchronize “visual” cortices across congenitally blind but not sighted individuals. J. Neurosci. 39, 8940–8948 (2019).31548238
25. Van Ackeren MJ Barbero FM Mattioni S Bottini R Collignon O Neuronal populations in the occipital cortex of the blind synchronize to the temporal dynamics of speech eLife 2018 7 e31640 10.7554/eLife.31640 29338838
Van Ackeren, M. J., Barbero, F. M., Mattioni, S., Bottini, R. & Collignon, O. Neuronal populations in the occipital cortex of the blind synchronize to the temporal dynamics of speech. eLife 7, e31640 (2018).29338838
26. Karns CM Dow MW Neville HJ Altered cross-modal processing in the primary auditory cortex of congenitally deaf adults: a visual-somatosensory fMRI study with a double-flash illusion J. Neurosci. 2012 32 9626 9638 10.1523/JNEUROSCI.6488-11.2012 22787048
Karns, C. M., Dow, M. W. & Neville, H. J. Altered cross-modal processing in the primary auditory cortex of congenitally deaf adults: a visual-somatosensory fMRI study with a double-flash illusion. J. Neurosci. 32, 9626–9638 (2012).22787048
27. Hunt DL Yamoah EN Krubitzer L Multisensory plasticity in congenitally deaf mice: how are cortical areas functionally specified? Neuroscience 2006 139 1507 1524 10.1016/j.neuroscience.2006.01.023 16529873
Hunt, D. L., Yamoah, E. N. & Krubitzer, L. Multisensory plasticity in congenitally deaf mice: how are cortical areas functionally specified? Neuroscience 139, 1507–1524 (2006).16529873
28. Meredith MA Lomber SG Somatosensory and visual crossmodal plasticity in the anterior auditory field of early-deaf cats Hearing Res. 2011 280 38 47 10.1016/j.heares.2011.02.004
Meredith, M. A. & Lomber, S. G. Somatosensory and visual crossmodal plasticity in the anterior auditory field of early-deaf cats. Hearing Res. 280, 38–47 (2011).
29. Shore SE Koehler S Oldakowski M Hughes LF Syed S Dorsal cochlear nucleus responses to somatosensory stimulation are enhanced after noise-induced hearing loss: trigeminal input to cochlear nucleus with deafness Eur. J. Neurosci. 2007 27 155 168 10.1111/j.1460-9568.2007.05983.x
Shore, S. E., Koehler, S., Oldakowski, M., Hughes, L. F. & Syed, S. Dorsal cochlear nucleus responses to somatosensory stimulation are enhanced after noise-induced hearing loss: trigeminal input to cochlear nucleus with deafness. Eur. J. Neurosci. 27, 155–168 (2007).
30. Zeng C Yang Z Shreve L Bledsoe S Shore S Somatosensory projections to cochlear nucleus are upregulated after unilateral deafness J. Neurosci. 2012 32 15791 15801 10.1523/JNEUROSCI.2598-12.2012 23136418
Zeng, C., Yang, Z., Shreve, L., Bledsoe, S. & Shore, S. Somatosensory projections to cochlear nucleus are upregulated after unilateral deafness. J. Neurosci. 32, 15791–15801 (2012).23136418
31. Cardin V Grin K Vinogradova V Manini B Crossmodal reorganisation in deafness: mechanisms for functional preservation and functional change Neurosci. Biobehav. Rev. 2020 113 227 237 10.1016/j.neubiorev.2020.03.019 32199886
Cardin, V., Grin, K., Vinogradova, V. & Manini, B. Crossmodal reorganisation in deafness: mechanisms for functional preservation and functional change. Neurosci. Biobehav. Rev. 113, 227–237 (2020).32199886
32. MacSweeney M Dissociating linguistic and nonlinguistic gestural communication in the brain NeuroImage 2004 22 1605 1618 10.1016/j.neuroimage.2004.03.015 15275917
MacSweeney, M. et al. Dissociating linguistic and nonlinguistic gestural communication in the brain. NeuroImage 22, 1605–1618 (2004).15275917
33. Benetti S Functional selectivity for face processing in the temporal voice area of early deaf individuals Proc. Natl Acad. Sci. 2017 114 E6437 E6446 10.1073/pnas.1618287114 28652333
Benetti, S. et al. Functional selectivity for face processing in the temporal voice area of early deaf individuals. Proc. Natl Acad. Sci. 114, E6437–E6446 (2017).28652333
34. Kral A Schroeder J-H Klinke R Engel AK Absence of cross-modal reorganization in the primary auditory cortex of congenitally deaf cats Exp. Brain Res. 2003 153 605 613 10.1007/s00221-003-1609-z 12961053
Kral, A., Schroeder, J.-H., Klinke, R. & Engel, A. K. Absence of cross-modal reorganization in the primary auditory cortex of congenitally deaf cats. Exp. Brain Res. 153, 605–613 (2003).12961053
35. Kral A Yusuf PA Land R Higher-order auditory areas in congenital deafness: top-down interactions and corticocortical decoupling Hearing Res. 2017 343 50 63 10.1016/j.heares.2016.08.017
Kral, A., Yusuf, P. A. & Land, R. Higher-order auditory areas in congenital deafness: top-down interactions and corticocortical decoupling. Hearing Res. 343, 50–63 (2017).
36. Berger C Kühne D Scheper V Kral A Congenital deafness affects deep layers in primary and secondary auditory cortex: BERGER et al J. Comp. Neurol. 2017 525 3110 3125 10.1002/cne.24267 28643417
Berger, C., Kühne, D., Scheper, V. & Kral, A. Congenital deafness affects deep layers in primary and secondary auditory cortex: BERGER et al. J. Comp. Neurol. 525, 3110–3125 (2017).28643417
37. Cardin, V. et al. Dissociating cognitive and sensory neural plasticity in human superior temporal cortex. Nat. Commun. 4, (2013).
38. Emmorey K McCullough S Mehta S Ponto LLB Grabowski TJ Sign language and pantomime production differentially engage frontal and parietal cortices Lang. Cogn. Process. 2011 26 878 901 10.1080/01690965.2010.492643 21909174
Emmorey, K., McCullough, S., Mehta, S., Ponto, L. L. B. & Grabowski, T. J. Sign language and pantomime production differentially engage frontal and parietal cortices. Lang. Cogn. Process. 26, 878–901 (2011).21909174
39. Fine I Finney EM Boynton GM Dobkins KR Comparing the effects of auditory deprivation and sign language within the auditory and visual cortex J. Cogn. Neurosci. 2005 17 1621 1637 10.1162/089892905774597173 16269101
Fine, I., Finney, E. M., Boynton, G. M. & Dobkins, K. R. Comparing the effects of auditory deprivation and sign language within the auditory and visual cortex. J. Cogn. Neurosci. 17, 1621–1637 (2005).16269101
40. Sadato N Cross-modal integration and plastic changes revealed by lip movement, random-dot motion and sign languages in the hearing and deaf Cereb. Cortex 2005 15 1113 1122 10.1093/cercor/bhh210 15563723
Sadato, N. et al. Cross-modal integration and plastic changes revealed by lip movement, random-dot motion and sign languages in the hearing and deaf. Cereb. Cortex 15, 1113–1122 (2005).15563723
41. Lomber SG Meredith MA Kral A Cross-modal plasticity in specific auditory cortices underlies visual compensations in the deaf Nat. Neurosci. 2010 13 1421 1427 10.1038/nn.2653 20935644
Lomber, S. G., Meredith, M. A. & Kral, A. Cross-modal plasticity in specific auditory cortices underlies visual compensations in the deaf. Nat. Neurosci. 13, 1421–1427 (2010).20935644
42. Deen B Koldewyn K Kanwisher N Saxe R Functional organization of social perception and cognition in the superior temporal sulcus Cereb. Cortex 2015 25 4596 4609 10.1093/cercor/bhv111 26048954
Deen, B., Koldewyn, K., Kanwisher, N. & Saxe, R. Functional organization of social perception and cognition in the superior temporal sulcus. Cereb. Cortex 25, 4596–4609 (2015).26048954
43. Fedorenko E Behr MK Kanwisher N Functional specificity for high-level linguistic processing in the human brain Proc. Natl Acad. Sci. Usa. 2011 108 16428 16433 10.1073/pnas.1112937108 21885736
Fedorenko, E., Behr, M. K. & Kanwisher, N. Functional specificity for high-level linguistic processing in the human brain. Proc. Natl Acad. Sci. Usa. 108, 16428–16433 (2011).21885736
44. Sueoka, Y., Paunov, A., Ivanova, A., Blank, I. A. & Fedorenko, E. The Language Network Reliably ‘Tracks’ Naturalistic Meaningful Non-Verbal Stimuli. 10.1101/2022.04.24.489316 (2022).
45. Sueoka, Y. et al. The language network reliably ‘tracks’ naturalistic meaningful non-verbal stimuli. Neurobiology of Language 1–64 10.1162/nol_a_00135 (2024).
46. Ivanova AA The language network is recruited but not required for nonverbal event semantics Neurobiol. Lang. 2021 2 176 201 10.1162/nol_a_00030
Ivanova, A. A. et al. The language network is recruited but not required for nonverbal event semantics. Neurobiol. Lang. 2, 176–201 (2021).
47. Blank H Davis MH Prediction errors but not sharpened signals simulate multivoxel fMRI patterns during speech perception PLoS Biol. 2016 14 e1002577 10.1371/journal.pbio.1002577 27846209
Blank, H. & Davis, M. H. Prediction errors but not sharpened signals simulate multivoxel fMRI patterns during speech perception. PLoS Biol. 14, e1002577 (2016).27846209
48. Newman AJ Supalla T Fernandez N Newport EL Bavelier D Neural systems supporting linguistic structure, linguistic experience, and symbolic communication in sign language and gesture Proc. Natl Acad. Sci. USA 2015 112 11684 11689 10.1073/pnas.1510527112 26283352
Newman, A. J., Supalla, T., Fernandez, N., Newport, E. L. & Bavelier, D. Neural systems supporting linguistic structure, linguistic experience, and symbolic communication in sign language and gesture. Proc. Natl Acad. Sci. USA 112, 11684–11689 (2015).26283352
49. Corbetta M Shulman GL Control of goal-directed and stimulus-driven attention in the brain Nat. Rev. Neurosci. 2002 3 201 215 10.1038/nrn755 11994752
Corbetta, M. & Shulman, G. L. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3, 201–215 (2002).11994752
50. Ding H Cross-modal activation of auditory regions during visuo-spatial working memory in early deafness Brain 2015 138 2750 2765 10.1093/brain/awv165 26070981
Ding, H. et al. Cross-modal activation of auditory regions during visuo-spatial working memory in early deafness. Brain 138, 2750–2765 (2015).26070981
51. Geng JJ Mangun GR Right temporoparietal junction activation by a salient contextual cue facilitates target discrimination NeuroImage 2011 54 594 601 10.1016/j.neuroimage.2010.08.025 20728548
Geng, J. J. & Mangun, G. R. Right temporoparietal junction activation by a salient contextual cue facilitates target discrimination. NeuroImage 54, 594–601 (2011).20728548
52. Geng JJ Vossel S Re-evaluating the role of TPJ in attentional control: contextual updating? Neurosci. Biobehav. Rev. 2013 37 2608 2620 10.1016/j.neubiorev.2013.08.010 23999082
Geng, J. J. & Vossel, S. Re-evaluating the role of TPJ in attentional control: contextual updating? Neurosci. Biobehav. Rev. 37, 2608–2620 (2013).23999082
53. Seymour JL Reorganization of neural systems mediating peripheral visual selective attention in the deaf: An optical imaging study Hearing Res. 2017 343 162 175 10.1016/j.heares.2016.09.007
Seymour, J. L. et al. Reorganization of neural systems mediating peripheral visual selective attention in the deaf: An optical imaging study. Hearing Res. 343, 162–175 (2017).
54. Campanella S Belin P Integrating face and voice in person perception Trends Cogn. Sci. 2007 11 535 543 10.1016/j.tics.2007.10.001 17997124
Campanella, S. & Belin, P. Integrating face and voice in person perception. Trends Cogn. Sci. 11, 535–543 (2007).17997124
55. Rennig J Beauchamp MS Free viewing of talking faces reveals mouth and eye preferring regions of the human superior temporal sulcus NeuroImage 2018 183 25 36 10.1016/j.neuroimage.2018.08.008 30092347
Rennig, J. & Beauchamp, M. S. Free viewing of talking faces reveals mouth and eye preferring regions of the human superior temporal sulcus. NeuroImage 183, 25–36 (2018).30092347
56. Puce A Perrett D Electrophysiology and brain imaging of biological motion Philos. Trans. R. Soc. Lond. B 2003 358 435 445 10.1098/rstb.2002.1221 12689371
Puce, A. & Perrett, D. Electrophysiology and brain imaging of biological motion. Philos. Trans. R. Soc. Lond. B 358, 435–445 (2003).12689371
57. Saxe R Xiao D-K Kovacs G Perrett DI Kanwisher N A region of right posterior superior temporal sulcus responds to observed intentional actions Neuropsychologia 2004 42 1435 1446 10.1016/j.neuropsychologia.2004.04.015 15246282
Saxe, R., Xiao, D.-K., Kovacs, G., Perrett, D. I. & Kanwisher, N. A region of right posterior superior temporal sulcus responds to observed intentional actions. Neuropsychologia 42, 1435–1446 (2004).15246282
58. Setti F A modality-independent proto-organization of human multisensory areas Nat. Hum. Behav. 2023 7 397 410 10.1038/s41562-022-01507-3 36646839
Setti, F. et al. A modality-independent proto-organization of human multisensory areas. Nat. Hum. Behav. 7, 397–410 (2023).36646839
59. Stojanoski B Cusack R Time to wave good-bye to phase scrambling: creating controlled scrambled images using diffeomorphic transformations J. Vis. 2014 14 6 6 10.1167/14.12.6 25301014
Stojanoski, B. & Cusack, R. Time to wave good-bye to phase scrambling: creating controlled scrambled images using diffeomorphic transformations. J. Vis. 14, 6–6 (2014).25301014
60. Kumar, M. et al. BrainIAK: the brain imaging analysis kit. Apert. Neuro. 1, 10.52294/31bb5b68-2184-411b-8c00-a1dacb61e1da (2021).
61. Kumar M BrainIAK tutorials: User-friendly learning materials for advanced fMRI analysis PLoS Comput Biol. 2020 16 e1007549 10.1371/journal.pcbi.1007549 31940340
Kumar, M. et al. BrainIAK tutorials: User-friendly learning materials for advanced fMRI analysis. PLoS Comput Biol. 16, e1007549 (2020).31940340
62. Esteban, O. et al. FMRIPrep: a robust preprocessing pipeline for functional MRI. 30 (2019).
63. Chang LJ Endogenous variation in ventromedial prefrontal cortex state dynamics during naturalistic viewing reflects affective experience Sci. Adv. 2021 7 eabf7129 10.1126/sciadv.abf7129 33893106
Chang, L. J. et al. Endogenous variation in ventromedial prefrontal cortex state dynamics during naturalistic viewing reflects affective experience. Sci. Adv. 7, eabf7129 (2021).33893106
64. Chen G Untangling the relatedness among correlations, part I: nonparametric approaches to inter-subject correlation analysis at the group level NeuroImage 2016 142 248 259 10.1016/j.neuroimage.2016.05.023 27195792
Chen, G. et al. Untangling the relatedness among correlations, part I: nonparametric approaches to inter-subject correlation analysis at the group level. NeuroImage 142, 248–259 (2016).27195792
65. Chen G Taylor PA Shin Y-W Reynolds RC Cox RW Untangling the relatedness among correlations, Part II: Inter-subject correlation group analysis through linear mixed-effects modeling NeuroImage 2017 147 825 840 10.1016/j.neuroimage.2016.08.029 27751943
Chen, G., Taylor, P. A., Shin, Y.-W., Reynolds, R. C. & Cox, R. W. Untangling the relatedness among correlations, Part II: Inter-subject correlation group analysis through linear mixed-effects modeling. NeuroImage 147, 825–840 (2017).27751943
66. Anderson, M. J. Permutational multivariate analysis of variance (PERMANOVA). in Wiley StatsRef: Statistics Reference Online (eds. Balakrishnan, N. et al.) 1–15, 10.1002/9781118445112.stat07841.(Wiley, 2017).
67. Schaefer A Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI Cereb. Cortex 2018 28 3095 3114 10.1093/cercor/bhx179 28981612
Schaefer, A. et al. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI. Cereb. Cortex 28, 3095–3114 (2018).28981612
68. Amunts K Mohlberg H Bludau S Zilles K Julich-Brain: A 3D probabilistic atlas of the human brain’s cytoarchitecture Science 2020 369 988 992 10.1126/science.abb4588 32732281
Amunts, K., Mohlberg, H., Bludau, S. & Zilles, K. Julich-Brain: A 3D probabilistic atlas of the human brain’s cytoarchitecture. Science 369, 988–992 (2020).32732281
69. Zachlod D Four new cytoarchitectonic areas surrounding the primary and early auditory cortex in human brains Cortex 2020 128 1 21 10.1016/j.cortex.2020.02.021 32298845
Zachlod, D. et al. Four new cytoarchitectonic areas surrounding the primary and early auditory cortex in human brains. Cortex 128, 1–21 (2020).32298845
