
==== Front
iScience
iScience
iScience
2589-0042
Elsevier

S2589-0042(24)02033-9
10.1016/j.isci.2024.110808
110808
Article
Highly compromised auditory spatial perception in aided congenitally hearing-impaired and rapid improvement with tactile technology
Snir Adi adisaxophone@gmail.com
134∗
Cieśla Katarzyna 123
Vekslar Rotem 1
Amedi Amir 1
1 The Baruch Ivcher Institute for Brain, Cognition, and Technology, The Baruch Ivcher School of Psychology, Reichman University, HaUniversita 8 Herzliya 461010, Israel
2 World Hearing Centre, Institute of Physiology and Pathology of Hearing, Mokra 17, 05-830 Kajetany, Nadarzyn, Poland
∗ Corresponding author adisaxophone@gmail.com
3 These authors contributed equally

4 Lead contact

25 8 2024
20 9 2024
25 8 2024
27 9 11080821 5 2024
11 7 2024
21 8 2024
© 2024 The Author(s)
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
Summary

Spatial understanding is a multisensory construct while hearing is the only natural sense enabling the simultaneous perception of the entire 3D space. To test whether such spatial understanding is dependent on auditory experience, we study congenitally hearing-impaired users of assistive devices. We apply an in-house technology, which, inspired by the auditory system, performs intensity-weighting to represent external spatial positions and motion on the fingertips. We see highly impaired auditory spatial capabilities for tracking moving sources, which based on the “critical periods” theory emphasizes the role of nature in sensory development. Meanwhile, for tactile and audio-tactile spatial motion perception, the hearing-impaired show performance similar to typically hearing individuals. The immediate availability of 360° external space representation through touch, despite the lack of such experience during the lifetime, points to the significant role of nurture in spatial perception development, and to its amodal character. The findings show promise toward advancing multisensory solutions for rehabilitation.

Graphical abstract

Highlights

• Highly impaired auditory spatial perception in congenitally hearing impaired (nature)

• Spatial tactile information significantly and rapidly improves results (nurture)

• Lack of auditory spatial experience does not impede multimodal spatial understanding

• Results show significant potential for the multisensory rehabilitation of spatial acuity

Neuroscience; Sensory neuroscience; Cognitive neuroscience; Techniques in neuroscience

Subject areas

Neuroscience
Sensory neuroscience
Cognitive neuroscience
Techniques in neuroscience
Published: August 25, 2024
==== Body
pmcIntroduction

Imagine a world where you can perceive sounds, perhaps even recognize what they are, but you find it impossible to pinpoint where exactly they are coming from. When someone calls you from the left, you turn right; or even worse, a car approaches you from out of your sight, you hear the noise, but cannot use the basic instinct to avoid coming in contact with it. These types of everyday challenges are constantly faced by the hearing-impaired population, including experienced bilateral cochlear implant users.1,2,3,4,5

When all sensory modalities are available, they collectively provide information about the 3D space, thus rendering spatial perception an amodal or multisensory experience.6,7 Audition, however, is the only sensory modality capable of perceiving external sources from all angles simultaneously (including behind the head) without moving the head or the body.8 Meanwhile, vision is only frontally oriented, and the tactile system (traditionally) only represents information within the peripersonal space, i.e., within reach. Does this entail that lack of optimal spatial auditory experience prevents the proper development of 3D spatial representation?

According to certain seminal works, sensory-specific information must be acquired early in life during “critical periods” for certain abilities to properly develop.9,10 Some authors also indicate that early sensory exposure is necessary for multisensory processes to occur.11,12,13 Findings in deaf or hearing-impaired individuals, including those equipped with hearing devices (such as hearing aids/HAs or cochlear implants, CIs), who perform poorly in spatial localization tasks, even following multiple years of access to binaural information seem to confirm that assumption for the auditory system function. In other words, the poor performance of the hearing aid/cochlear implant users seems to underscore the role of nature vs. nurture in sensory development.14 This in relation to the classical nature vs. nurture debate concerning the relative contribution of biological/genetic factors vs. the environment (i.e., life experiences and learning), both influencing the individual’s development and behavior.

Auditory localization is based on binaural cues derived from constantly performed comparisons of the Interaural Time Differences (ITDs) and the interaural level differences (ILDs) of the sources arriving at the two ears. Additional monaural cues are available due to the individual’s shape of the ear, head, and torso.15 To calculate the auditory source position accurately, the auditory system must perform calculations at extremely high speeds, and even more so when sources are in motion.8,16

Hearing impairment affects sound localization by either limiting the perceived frequency ranges or by distorting the incoming sounds, and thus also the binaural cues. In addition, auditory devices (hearing aids, cochlear implants), optimized for speech comprehension, do not preserve the localization cues well, in part due to the specific sound encoding algorithms, signal compression, and gain adjustments.5,17,18,19 Furthermore, the post-treatment training for the hearing impaired focuses mainly on speech comprehension.20,21

Furthermore, spatial localization abilities of the hearing impaired are often tested using non-ecological static sounds positioned in the frontal field,1,2,4,5,22 with only a few recent studies taking into account the entire surrounding 360° space, including the area behind the head,23,24,25 as well as dynamic sound scenes.3,26,27

Given these limitations, and the fact that people who are born with hearing impairments only have experience of perceiving space in a compromised manner, can an alternative sensory modality be considered to represent spatial information in a manner similar to the auditory system in this population?

In our recently published work, we describe our in-house built technology representing 3D spatial motion, the touch-motion algorithm (TMA28). Similar to the weighting performed by the auditory system to decode source locations, TMA reproduces spatial positions via level-weighting on multiple vibrotactile actuators (emitted to four fingers). As with hearing, source locations are reproduced with the head in the center as a reference point. We show that typically hearing individuals can use weighted tactile stimulation to represent spatial motion with accuracy comparable to that of a corresponding auditory task, along with enhanced performance for audio-tactile integration in background auditory noise.

This line of work was inspired by the animal kingdom. Elephants, in particular, are capable of perceiving sound information, as well as localizing its source, using mechanoreceptors on their feet.29 At the same time, some species of fish are equipped with a lateral line of epithelial cells on the sides of the body which encode fluid motion in water. It is speculated that the inner ear in mammals (and especially the sensory cells themselves) developed from a vestibular organ stemming from the lateral line.30 Furthermore, the auditory and the tactile systems in both mammals and humans alike, are densely connected at all levels of the central nervous system, and both are capable of encoding vibrations in a shared frequency range of 30-1000Hz through mechanoreceptors31,32,33,34,35,36 (with a common ancestral lineage claimed between certain tactile mechanoreceptors and cochlear hair cells37).

In the current study, we further expand our investigation into congenitally hearing-impaired individuals. This population is of particular interest due to their presumably impaired auditory spatial understanding, including lacking access to extrapersonal sensory information outside of the visual field (mainly behind the head). Specifically, our experiment measures localization capabilities of auditory, tactile and paired auditory-tactile sources moving on a 360° azimuthal plane in a group of bilateral cochlear implant users with congenital hearing loss and early fitting of one cochlear implant. In addition, we test single subjects with one cochlear implant, a bimodal cochlear implant-hearing aid solution or bilateral hearing aids. Our multisensory approach enables us to investigate their 3D spatial perception regardless of their auditory capabilities.

We propose this new approach, based on studies that applied auditory and tactile devices (sensory substitution devices, SSD) in congenitally blind individuals. The authors showed that certain visual skills, such as an object or color recognition are learnable in adulthood following hours of training through different sensory modalities (indicating the role of nurture vs. nature in the development of sensory functionalities despite lack of early experience,9,38 see also precursor studies in this field by Bach-y-Rita 196939).

Based on the existing research in the hearing-impaired population, we hypothesize that the cochlear implant/hearing aid users will display compromised auditory localization abilities also for 3D moving sources, as compared to those with typical hearing (TH). This, assumes the critical role of nature in the development of spatial perception conditional upon access to natural binaural cues. For tactile spatial localization in the hearing-impaired group, we posit that if an early experience of 3D space through hearing is necessary for the development of 3D spatial perception in general, then performance in the tactile task will also be compromised. At the same time, if the tactile modality is capable of performing the new 3D task, this will strengthen the role of nurture in building a 3D space representation, and emphasize its amodal character. In addition to the main goals of the study, we ask the participants to perform the same spatial localization task with both auditory and tactile cues available (congruent in space and in content). This in order to mirror a possible real-world scenario, where we continuously encounter information from multiple senses. For the hearing-impaired population, auditory information is often noisy (unreliable), yet they depend on it throughout their lifetime. At the same time, the 3D tactile input is new and untrained. Based on their scores and the reported subjective experience, we investigate whether they will mainly rely on their auditory sense in the multisensory task or turn to the novel (and potentially more reliable) sensory modality.

Results

Spatial motion localization

In the spatial localization tasks, the participants were asked to localize stimuli moving on a 360° azimuth around them. Specifically, they had to indicate the start and end positions, as well as the direction of trajectories of moving sources. The task was either auditory (two audio conditions at the very beginning and at the very end; Audio-Baseline & Audio-Post), tactile (using touch motion algorithm, TMA; Tactile), or combining both sources of information (i.e., Audio-Tactile). TMA uses auditory-inspired features for tactile-enabled localization via intensity-weighted comparisons. Tactile inputs were emitted as “moving” vibrations onto four fingertips of two hands placed on the sides of the person. Responses in the tasks were given verbally (e.g., “The sound moved from left to right in front of me”). The mean distance in angle between the perceived and the actual stimulus midpoint was converted to a score on a scale of 0–1 (perfect response: 0° error = 1; maximum error of 180° = 0). We report the scores as both Arbitrary Units (A.U.; 0–1) and as errors in degrees.

Hearing impaired individuals perform significantly worse than the typically hearing controls in auditory spatial motion localization

Scores in the Audio-Baseline condition

Users of bilateral cochlear implants (N = 10) obtained the following scores in the Audio-Baseline test condition: mean (0.58 ± 0.018 A.U., 75.6° error), median (0.62, IQR = 0.50). When analyzed jointly, the whole hearing impaired group (N = 19) had very similar scores; Audio-Baseline mean score (0.58 ± 0.19 A.U.; 75.6° error), median (0.57, IQR = 0.38). The typically hearing individuals (N = 29) obtained the following results in the same task: Audio-Baseline mean score (0.90 ± 0.19 A.U.; 18° error), median (1, IQR = 0.25). All Audio-Baseline scores were significantly above chance (Bonferroni adjusted p < 0.006; Wilcoxon signed-rank test). The bilateral cochlear implant users showed significantly lower auditory scores, as compared to the typically hearing group in the Audio-Baseline condition (z = 16.8, p < 0.001; Mann Whitney tests; large effect size, Cohen’s d = 1.3, power 99%, Bonferroni adjusted p < 0.004), and the same was found for the whole hearing impaired group (z = 19.9, p < 0.001). See also Figure 1.Figure 1 Experimental apparatus and Audio-Baseline scores

Auditory spatial motion localization in the Audio-Baseline (A1) test condition.

(A) Experimental set-up with 97 speakers on the walls and on the ceiling, with tactile devices placed on both sides of the participants.

(B) A scheme representing score calculation in arbitrary units (A.U.) and error in degrees.

(C) Group scores in the Audio-Baseline condition (A1) in congenitally hearing impaired users of bilateral cochlear implants (N = 10)), in comparison to the typically hearing individuals (N = 29; Wilcoxon signed-ranks test; ∗∗∗p < 0.001, ∗∗p < 0.01, ∗p < 0.05; Wilcoxon signed-rank tests). Scores of the bilateral cochlear implant users (bi-CI) and typically hearing individuals (TH) are represented in A.U. as scatterplots of individual data, as well as the minimum, first quartile, median, third quartile, and maximum; individual scores of other hearing device users (Other CI/HA; N = 9) are represented as symbols on the right side; CI - cochlear implant, HA - hearing aid.

Scores in the Audio-Post condition

In the Audio-Post task, bilateral cochlear implant users received the following scores: Audio-Post mean (0.55 ± 0.18 A.U.; 81° error), Audio-Post median (0.62, IQR = 0.59). The scores of all cochlear implant/hearing aid users together were in a similar range: mean (0.59 ± 0.012 A.U.; 73.8° error), median (0.62, IQR = 0.50). For the typically hearing individuals, the scores were: mean (0.91 ± 0.19 A.U.; 16.2° error), median (1, IQR = 0). All scores were significantly above chance (Bonferroni adjusted p < 0.006; Wilcoxon signed-rank), except for the mean Audio-Post scores in bilateral cochlear implant users (p = 0.01). Furthermore, in bilateral cochlear implant users the results were significantly lower than those found in the typical hearing (z = 18.12, p < 0.01; Mann Whitney tests; Cohen’s d = 1.3, power = 99%, Bonferroni adjusted p < 0.004), and the same was found for all cochlear implant/hearing aid users jointly (z = 20.1, p < 0.001).

No statistically significant differences were found between the Audio-Baseline and the Audio-Post scores in either of the groups (Mann-Whitney test, p > 0.004).

Hearing-impaired individuals show significantly higher results when performing the task through touch, as compared to audio only

In the bilateral cochlear implant users, the Tactile scores were as follows: mean (0.78 ± 0.16 A.U.; mean 39.6° error), median (0.875, IQR = 0.25), and were significantly higher than the Audio-Baseline results in this group (z = −7.35, p < 0.001, Wilcoxon signed-ranks test; Bonferroni adjusted p < 0.004). When analyzed together, the group of all cochlear implant/hearing aid users (N = 19) obtained the following very similar results in the Tactile condition: mean score (0.79 ± 0.011 A.U.; 37.8° error), median (0.87, IQR = 0.25), and these were also significantly higher than the Audio-Baseline results (z = 11.4, p < 0.001; large effect size, Cohen’s d = 0.85, power = 98%; Bonferroni adjusted p < 0.004). Within the typically hearing group, the Tactile results were as follows: mean (0.86 ± 0.01 A.U.; 25.2° error), median (1, IQR = 0.25), and were significantly lower than their Audio-Baseline results (z = 5.3; p < 0.001; Wilcoxon signed-rank test; Bonferroni adjusted p < 0.004). All Tactile scores were significantly above chance (Bonferroni adjusted p < 0.006; Wilcoxon signed-rank). When the groups were compared with one another, Mann-Whitney tests showed significantly lower scores in the participants with hearing loss, as compared to the typically hearing subjects in the Tactile test, both for bilateral cochlear implant users (z = 4.11, p < 0.001; Bonferroni adjusted p < 0.004) and all the hearing-impaired subjects jointly (z = 4.15, p < 0.001; Bonferroni adjusted p < 0.004). See also Figure 2.Figure 2 Spatial motion localization task results

Audio-Baseline (A1), Tactile (T), and Audio-Tactile (AT) spatial motion localization scores.

(A) Experimental set-up (from above) with fingertips inserted in the two vibrotactile devices on the sides of the participant, and an example trajectory of a moving stimulus; (B) group scores in the Audio-Baseline and Tactile conditions in bilateral cochlear implant users (N = 10) in comparison to the typically hearing individuals (N = 29), (C) group scores in the Audio-Baseline and Audio-Tactile conditions in congenitally hearing impaired bilateral cochlear implant users in comparison to the typically hearing individuals, (D) group scores in the Tactile and Audio-Tactile conditions in bilateral cochlear implant users in comparison to the typically hearing individuals. Comparisons using Wilcoxon signed-rank tests (∗∗∗p < 0.001, ∗∗p < 0.01, ∗p < 0.05); scores are represented in A.U. as scatterplots of individual data, as well as the minimum, first quartile, median, third quartile, and maximum; individual scores of other hearing device users are represented as symbols on the right side (Other CI/HA; N = 9); CI - cochlear implant, HA - hearing aid.

Hearing impaired individuals show significantly higher results when performing a multisensory task, as compared to audio only but not when compared to tactile only

During the Audio-Tactile test condition, bilateral cochlear implant users had the following results: mean score 0.79 ± 0.16 A.U., 37.8° error, median (0.87, IQR = 0.25). The scores were significantly higher than their Audio-Baseline scores (Z = 8.27, p < 0.001; Bonferroni adjusted p < 0.004). The results of the entire hearing-impaired group were almost the same, i.e., mean score (0.81 ± 0.011 A.U.; 34.2° error); median (0.87, IQR = 0.25) and were also found significantly higher than their Audio-Baseline scores (z = 13.15, p < 0.001; Bonferroni adjusted p < 0.004).

The group of typically hearing had the following results: mean (0.93 ± 0.006 A.U.; 12.6° error), median (1, IQR = 0.25), and their Audio-Tactile results were not different from their Audio-Baseline scores (p > 0.004). Mann Whitney tests showed significantly lower scores in all participants with hearing loss (combined bilateral cochlear implant and hearing aid users), as compared to the typically hearing subjects in the Audio-Tactile test condition (z = 9.1, p < 0.001; Bonferroni adjusted p < 0.004). The same was shown for the group of bilateral cochlear implant users separately (z = 9.1, p < 0.001; Bonferroni adjusted p < 0.004). All Tactile scores were significantly above chance (Bonferroni adjusted p < 0.006; Wilcoxon signed-rank test).

The Audio-Tactile results in the whole hearing impaired group and in the users of bilateral cochlear implants separately were not found statistically significantly different from their results in the Tactile condition (p > 0.004). At the same time, in the typically hearing subjects, the Audio-Tactile results were significantly higher than their Tactile results z = −3.12, p = 0.002; Bonferroni adjusted p < 0.004). See Figure 2.

Individual results of hearing-impaired subjects show a similar pattern between test conditions

Figure 3 represents the results of single hearing-impaired subjects, showing a very similar pattern in performance across all individuals. All participants performed slightly above the chance level in the two audio tasks, and all showed higher results in the Tactile and Audio-Tactile tasks.Figure 3 Single subject scores of the hearing-impaired individuals

Single subject results in the spatial localization task in the hearing impaired individuals, for all test conditions. ; HA - hearing aid, BC - bone conduction aid. Scores are represented in arbitrary units (A.U.).

Subjective questionnaires following motion localization tasks

The subjective questionnaire following participation in the motion localization tasks revealed some of the participants’ experiences. Questions regarding the Audio-Tactile task specifically showed that while the two sensory inputs felt equally connected and easy to combine for both the typically hearing and the hearing impaired, the latter group found the tactile input significantly more helpful for spatial localization (Wilcoxon ranked-sum test, p < 0.001). See Figure 4 (see also Table S1).Figure 4 Subjective questionnaire responses referring to the Audio-Tactile spatial localization condition

Responses to questions regarding subjective experience in the audio-tactile spatial localization task; TH - typically hearing (N = 29), biCI - bilateral cochlear implant users (N = 10), All CI/HA - the whole group of CI and hearing aid (HA; N = 19) users; biCI is part of the whole hearing impaired group. VAS - Visual analog scale.

See also Table S1.

Correlation analysis

In the group of bilateral cochlear implant users (N = 10) there was a negative correlation found between the age at fitting of both devices and the results in the Audio-Baseline test condition in the motion localization task (for the first cochlear implant: Kendall’s r = −0.53, p = 0.0.019; for the second cochlear implant: Kendall’s r = −0.56, p = 0.009). This result shows that an early implantation, and especially of the second implant can be related to better auditory spatial localization (see Figure 5).Figure 5 Correlation between the age of cochlear implantation and auditory spatial scores

Correlation analysis (Kendall r) and best-fit regression line between the spatial motion localization scores in the Audio-Baseline task and the age at the fitting of two cochlear implants in bilateral cochlear implant users (N = 10). Scores are represented in arbitrary units (A.U.).

Additional psychological questionnaires

Several additional screening tests were applied. Since attention plays a significant role in spatial localization, all participants performed the Colour Trial Test (CTT), a standard tool for measuring sustained and divided attention. The hearing impaired group in addition filled in a questionnaire (APHAB and/or NCIQ) to assess their subjective hearing performance and the perceived hearing loss-related handicap in everyday life.

Color trails test (CTT)

In the hearing-impaired subjects (3 males, 13 females, age 29.13 ± 11.5; combined cochlear implant and hearing aid users) the meantime of performing the CTT2 task was 71.06 ± 19.35 s, and no misses or corrections were recorded. In the typically hearing subjects (7 males, 19 females, age 23.37 ± 1.4) the mean time for the same task was 74.73 ± 16.73 s. There was no significant difference found in performance between groups (Wilcoxon rank-sum test: W = 231, p-value = 0.3362). These screening results show that participants' performance falls within the expected average range, indicating no issues with sustained or divided attention in either group.40

Nijmegen cochlear implant questionnaire (NCIQ)

The reported scores for the NCIQ questionnaire were found to be similar to those shown by other research groups for cochlear implant users.41,42,43 As an example, the average results for Basic Sound Perception, a scale referring to hearing sounds such as the phone, car, someone calling, and so forth, were: mean ± SD = 60.58 ± 17.75 (maximum 100 points). For the Advanced Sound Perception scale, referring to situations such as having a conversation, enjoying music, recognizing the gender of the speaker, and so forth, were: mean ± SD = 77.36 = 5 ± 13.7 (maximum 100 points), indicating medium-high performance. The remaining results are presented in Table S3.

Abbreviated profile of hearing handicap questionnaire (APHAB)

The revealed APHAB scores indicate that the participants varied with respect to the perceived hearing-related handicap, both with and without the hearing aid. However, they all showed a benefit of using a hearing aid in terms of Ease of Communication (i.e., communicating at home, in a public space), as well as in acoustic situations with reverbnation (e.g., in a classroom or a theater; except for RHALCI024) or Background Noise (e.g., when there are multiple talkers). In the scale Aversiveness, they all reported that the handicap was higher with the hearing aid turned on. This scale refers to situations such as traffic noise, sudden alarm sound, construction noise, and so forth. Most probably, these sounds are not available to the participants without a hearing aid. Table S2 shows the results of five single subjects using either two hearing aids or one hearing aid and one cochlear implant.

Discussion

Auditory spatial capabilities are severely impaired in hearing-impaired subjects, including bilateral cochlear implant users

Our results are in agreement with and further contribute to the existing research on sound localization capabilities in bilateral cochlear implant users and other hearing-impaired individuals.1,2,4,18,22 Even though some prior research maintains that people with bilateral cochlear implants can perform basic localization tasks better than those with unilateral cochlear implants, here we show that this population still has impaired spatial capabilities, and specifically when encountered with a real-world task involving surrounding sounds in motion.5,44,45 Bilateral cochlear implant users perform the auditory task only slightly above chance (at approximately 60%; the large effect size also indicates that 92% of the hearing impaired group in general had lower scores than the typically hearing subjects). Prior research on auditory motion perception in this population is scarce but shows that it can be more challenging than localizing static sources.3,46 Regarding unilateral cochlear implant users and bilateral hearing aid users, we show similar levels of auditory performance among these in our complex setting (cf. a similar observation for localizing static sources in Dorman et al., 20162).

The fact that auditory localization remains impaired despite years of auditory experience points toward the importance of nature for the proper establishment of auditory spatial representation. This may be a result of only partial access to sound level cues (ILDs) and the severely obstructed temporal cues (ITDs47,48,49), both essential for successful sound localization. The role of nature is further emphasized in the correlation between the age of fitting of the second cochlear implant in particular and auditory scores in our group of bilateral cochlear implant users.50 With both cochlear implants fitted early in life, a binaural model can be established and further refined with experience. At the same time, with the second cochlear implant fitted later in life (in most participants in the current study when they were teenagers), the spatial maps developed in a suboptimal manner. Indeed, early auditory deprivation in one ear, and prolonged stimulation in the other ear, have been shown to result in maladaptive aural preferences.14,51 Further investigation into the bilateral cochlear implant users population is nevertheless needed in order to assess the potential of dedicated (e.g., audio-tactile) interventions for auditory recalibration which could prove effective toward the development of auditory spatial abilities later in life (see similar discussion regarding blind in e.g., Gori, 201552; Bruns & Röder, 20236).

Newly acquired tactile ability for spatial localization

While the auditory localization of 360° spatial motion was impaired among the hearing-impaired individuals (both bilateral cochlear implant users and others), they achieved tactile accuracy nearly as high as the typically hearing group (scores of 0.82 vs. 0.84), which was also close to the typically hearing auditory performance (90%). The ability to perform a spatial localization task with such high accuracy through an alternative sensory modality, learned in adulthood, points to the role of nurture in the development of spatial representation.

This result also aligns with the broader framework of sensory substitution devices (SSD) and training programs which enable the establishment of a novel connection between a computation and an atypical sensory modality39,53,54,55,56; and further indicates the brain’s ability to use novel information that was not introduced during the critical periods of development nor in evolution.9 Such rapid development of a new skill using another sense, with accuracy comparable to the life-long developed auditory modality is however rare, and a possible indication of spatial perception being amodal and highly malleable in nature6,7 (This is also in line with neuroimaging literature showing shared neuronal mechanisms in the parietal cortex for spatial attention and in MT/V5 for motion processing, regardless of the applied sensory modality57,58).

In addition, high tactile performance in hearing-impaired individuals demonstrates that early sensory deprivation may not necessarily hinder the development of spatial representation, and further appropriate intervention may enhance such skills. Similarly, studies in congenitally blind individuals show effects of training leading to the acquisition of typically “visual” functions such as, e.g., navigation through vibrotactile inputs from an electronic walking cane55 or face recognition through specifically developed soundscapes.53,59 Research using SSDs in people with partially impaired senses using assistive technologies, as in the current study, is significantly more scarce.

Multisensory effects in the spatial localization task

Our main findings for the multisensory task (Audio-Tactile) indicate a common strategy applied by the typically hearing individuals and the hearing-impaired group. For the typically hearing Audio-Tactile scores were on par with the auditory scores, and for the hearing impaired with the Tactile scores, in both cases the higher performing sensory modality. At the same time, we show that multisensory scores exceeded auditory scores in the hearing-impaired group, and tactile scores in the typically hearing group, respectively.

In everyday situations, when exposed to simultaneous multisensory inputs, the optimal strategy adopted to complete a task depends on a number of factors (e.g., task conditions, temporal and spatial relations between the inputs, attention, prior knowledge, age, sensory status, and so forth), and can also change dynamically during task performance. One possible strategy might be to try to integrate all the incoming inputs (“binding tendency,” Odegaard et al. 201760), and another to rely more heavily on one specific sensory modality (either the one that is dominant for a given function as learned through experience or the one that is more reliable and/or less noisy in the specific context, see e.g., Hecht et al. 200961: “modality appropriateness hypothesis” vs. Bayesian causal inference models, respectively).

While in typically hearing it was to be expected that the highly functional auditory system would be prioritized for the spatial localization task, the case of the hearing-impaired individuals is different. The hearing-impaired group chooses the sensory input which is utterly new and untrained for 360° spatial localization, over one with which they have had experience with throughout their lifetime (but learned to be unreliable). This finding seems to abide by the theory claiming that sensory reliability is key to multisensory processing (Rhode et al. 2016).

These speculations are further informed by the subjective responses of the participants, where the Tactile and the Audio-Tactile spatial tasks felt significantly easier to the individuals with hearing impairments as compared to the typically hearing (for whom the auditory tasks were, as expected, reported as more difficult). The hearing impaired also found the tactile inputs much more helpful in localizing sounds than the typically hearing control group. In fact, using vibrotactile cues may be already familiar to hearing-impaired individuals, such as for example when touching speaker membranes while listening to music.62

Other existing findings in cochlear implant users are scarce and mixed with respect to multisensory effects in various tasks. In unilateral cochlear implant users, some works show no benefit of adding visual cues to an auditory vertical localization task,22 as well as impaired audio-tactile integration (parchment skin illusion) in cases of prolonged deafness (although accuracy improved with longer device use63,64; Guillemot & Champoux, 2014). Others indicate typical visual attention capture in bilateral cochlear implant users65,66 and efficient integration of simple audio-tactile stimuli in both congenitally and late unilateral/bilateral cochlear implant recipients, leading to faster reaction times.67

Similarities between auditory and tactile processing and subjective experiences

The use of specifically developed tactile vibrations in our study may have been critical for obtaining the observed high results in the tactile and audio-tactile tasks. Prior research using more basic set-ups has shown that vibrotactile stimulation can be used for improved speech comprehension,68,69,70,71 localization of simple static sources,72,73 and music perception,74 both in typically hearing individuals and in the hearing impaired/deaf subjects. We assume that this intuitive audio-tactile learning is due to the numerous similarities between these two senses, which is also the reason why the outcomes are more immediate, as compared to other multisensory solutions (see introduction; cf. long training regimes for audiovisual and visuo-tactile set-ups39,53,54,56).

In the current study, another parallel between the two sensory modalities can be drawn based on the subjective responses of the participants. When asked whether they were imagining/visualizing the tactile experience, 70% of typically hearing and 60% hearing hearing-impaired participants said “yes” (97% and 80% for the auditory stimuli, respectively). While it is a known phenomenon for auditory motion perception,16 this result indicates a possible development of the extrapersonal perception of tactile moving sources in some participants as well. This contributes to the general debate on whether objects rendered through SSDs can be perceived as existing directly in the external space, similar to the way natural senses operate (cf. distal attribution55,75).

Rehabilitation outlooks

We did not see significant improvement in the auditory motion localization (Audio-Post vs. Audio-Baseline) following a brief introduction of multisensory and tactile cues. However, we believe that applying longer dedicated training protocols could help calibrate auditory perception (see similar suggestions in Gori et al., 2014,73 Fletcher, Cunningham & Mills, 202072). This considering that a capability to adapt to experimentally altered binaural cues (e.g., through ear plugging) has been shown in a row of human and animal studies, as well as improved auditory localization capabilities after acquiring experience with hearing devices.4,76,77 Successful multisensory through audiovisual training for spatial perception has already been suggested66,78 (Aurelie 2022). The tactile sense handicap is that it can provide cues regarding the whole 360° azimuth, unlike vision, without head or body motion. In addition, our specific solution allows for an alignment of the two sensory inputs in terms of content, spatial movement, and timing. An audio-tactile intervention could potentially contribute to accelerating rehabilitation after a cochlear implant or hearing aid fitting, by refining the relationship between the various incoming auditory signals (e.g., electrical vs. acoustic, or between two cochlear implants), as well as their relationship to spatial locations.

Nature vs. nurture in light of our findings

Our findings support the role of both nature and nurture in the development of spatial understanding. For auditory spatial perception specifically, it seems that access to natural (or close to natural) cues is required to develop proper auditory spatial capabilities. Yet by using tactile means, we show that congenitally hearing impaired are still capable of representing 360° space. Furthermore, we show the rapid availability of the tactile sense in performing a task associated with the auditory sense via technologically mediated sensory input. These findings have implications for neuroscience and are crucial for the further development of sensory rehabilitation for populations with hearing/sensory impairments.

Limitations of the study

Our current work made use of one specific setup, with hands positioned to the sides of the body and spatially aligned with the auditory layout. Such a configuration was found most intuitive for the participants. The importance of body and limb posture has been found in multiple other works investigating tactile perception and perception of space. One series of studies investigated the applicability of a vibrotactile device allowing blind individuals to navigate in space using vibration feedback corresponding to the distance of visual objects. The authors showed, among others, that for optimal performance, the arm has to be aligned with the object (Meidenbaum et al. 2014). Prior research has also shown that it is mainly the relative arm posture (and not the actual anatomical location of inputs on the arm) that determines spatial decisions about the perceived tactile sensations.79 Furthermore, it was frequently shown that crossing the arms can impact one’s ability to perform temporal order judgments of vibrotactile inputs on the two hands, as well as tactile motion perception (both aspects relevant for the spatial tactile task performance in the current study80). Further investigation is needed with our specific setup to elucidate the role of the position of hands in relation to the body for tactile spatial localization. Furthermore, only one subgroup of the hearing impaired was homogeneous (10 users of bilateral cochlear implants), while the rest of the participants used various assistive device combinations: 3 were equipped with one cochlear implant and one hearing aid, 4 had one cochlear implant and the other ear was effectively deaf, and 2 wore bilateral hearing aids. While we did not see any differences in performance between members of these subpopulations (through visual inspection; see Figures 1, 2, and 3), more specific conclusions could arise by investigating these subgroups separately and with more participants. It would also be valuable to test the effects of variables, such as age at onset of hearing loss and its severity, duration of hearing loss, and cochlear implant/hearing aid use, on spatial localization skills, both through audition and touch.

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Dr Adi Snir (adisaxophone@gmail.com).

Materials availability

This study did not generate new unique materials.

Data and code availability

• Data and code reported in this article will be shared by the lead contact upon request.

• All original code reported in this article will be shared by the lead contact upon request.

• Any additional information required to reanalyze the data reported in this article is available from the lead contact upon request.

Acknowledgments

This project has received funding from the 10.13039/501100007601 European Union's Horizon 2020 research and innovation program under grant agreement No 101017884 , the 10.13039/501100000781 European Research Council under the 10.13039/501100007601 European Union's Horizon 2020 research and innovation program grant agreement No 773121 , and the 10.13039/501100000781 European Research Council proof of concept grant (project acronym TouchingSpace360 ). We would like to thank Oran Goral for his support with the illustrations, and Uri Barak for assisting with the statistical analysis.

Author contributions

Writing-original draft: AS and KC; writing - review and editing programming: AS, KC, and AA; software development: AS; methodology development: AS and KC; conceptualization: AS, KC, and AA; supervision: AS, KC, and AA; project administration: AS, KC, and RV; investigation: AS, KC, and RV; visualization: AS and RV; data curation: RV; resources: AA.

Declaration of interests

We declare a patent has been filed based on the technology featured in the current article.

The author A.A. is a co-founder and a member of the scientific advisory board of Remepy.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER	
Max MSP (Version 8.15)	Cycling 74	https://cycling74.com/	
Spat Library (Version 5.2)	Institute for Research and Coordination in Acoustics/Music (IRCAM)	https://forum.ircam.fr/projects/detail/spat/	
Touch-Motion Algorithm (TMA)	BCT Institute, Reichman University (Dr Adi Snir)	DOI: j.isci.2024.109820
Patent: WO2023095141A1	
SPSS Statistics ver. 29	IBM	https://www.ibm.com/products/spss-statistics	
Python ver. 3.12.2	Python Software Foundation	https://www.python.org/	
UMIK-1 (measurement microphone)	MiniDSP	https://www.minidsp.com/products/acoustic-measurement/umik-1	
Control 23-1L (passive loudspeakers)	JBL	https://jblpro.com/en/products/jbl-professional-control-23-1	
DCi 8|600DA (amplifier)	Crown Audio	https://www.crownaudio.com/en/products/dci-8-600da	
VAS (vibrotactile device)	Neurodevice/World Hearing Center (Dr Tomasz Wolak, Dr Katarzyna Ciesla)	https://doi.org/10.3233/RNN-190898	

Experimental model and study participants details

The study was approved by the Reichman University Institutional Review Board (approval no 2022123) and conformed to the 2013 Helsinki Declaration. All participants signed an Informed Consent for participating in the study and were financially compensated for their participation. They were recruited through social media and with the help of previous participants.

Nineteen (19) people with bilateral sensorineural hearing loss were included in the study (16 female, 3 males; age 26.5 ± 8.09). Ten (10) were users of two cochlear implants (8 female, 2 males; age 27 ± 6.4; number of years of using two cochlear implants: Mean ± SD = 13.45 ± 4.4), three (3) were users of one cochlear implant and one hearing aid (bimodal stimulation), four (4) were users of a cochlear implant in one ear with the other ear deaf (>90dB HL for 0.25-8kHz range) and two (2) were users of two hearing aids. Hearing loss was of various etiology, in all participants diagnosed at birth (17/19) or within the first 5 years of life. None of the participants except for one (EG) had tinnitus. For the 3 users of a cochlear implant and a hearing aid, in the ear with the hearing aid the aided hearing thresholds for 250Hz-8kHz were 50–90 dB (EY), 30-70dB (YL), 40-80dB (TM); one user of two hearing aids had a mean binaural aided hearing threshold of 80–120 dB (RS), and one user of two bone-conduction devices had 20-70 dB (EG). All participants with hearing aids had a sloping hearing loss. Most users of cochlear implants got their first cochlear implant early in life, i.e., until the age of 3 and the second implant/a hearing aid before the age of 15 years. The control group consisted of 29 individuals (22 female; 23.63 + 1.16; hereafter as typically hearing). Randomly selected 22 people from the control group had a tonal audiometry test using an audiometer MAICO MA-51, with a staircase procedure of 5 dB up- 10 dB down (in a soundbooth at the CANLAB laboratory of the Reichman University), and had normal hearing (<20dB for 250 Hz-8kHz). The exclusion criteria for the study were neurological or psychiatric diseases, attention deficits, and tactile sensory issues. All subjects were of European and/or Middle Eastern descent, as reported during an initial interview (see further details in Tables S4 and S5 in Supplementary Materials).

Method details

Apparatus

The experiment was conducted in a sound-treated cube-shaped room (4 × 4 meter), equipped with 97 loudspeakers (JBL Control 23-1L, powered by 13 Dante-enabled amplifiers, Crown Audio DCi 8|600DA) mounted on the walls and ceiling. The loudspeakers were organized in three horizontal rings, 24 loudspeakers each, at heights of 48cm, 148cm, 248cm from the ground and an azimuthal distance of 15° between adjacent speakers. The remaining twenty-five loudspeakers were set up on the ceiling in a 5x5 grid.

Auditory “moving” stimuli were decoded using an AllRad Ambisonics decoder (12th order) using the Spat5 library (version 5.2; Institut de Recherche et Coordination Acoustique/Musique, IRCAM) in the MaxMSP coding environment.

For the tactile stimulation, two in-house developed devices (VAS boxes) were used, each containing two piezoelectric plates to deliver vibrotactile stimulation on two fingertips (index and middle finger) of both hands. The devices were sound-proofed and contained two silicone slits through which each finger was inserted and placed on top of a piezoelectric actuator.71,81

Each of the four VAS actuators (four fingers) corresponded to a corner of the room (front-left, front-right, back-left, back-right), emphasizing the correspondence between hand positions and the external space (more details in: Snir & Ciesla, 202428). An author of the manuscript (AS) developed an algorithm, TMA (Tactile Motion Algorithm), which decoded virtual positions of sounds in 360° space to four vibrotactile actuators of the VAS device. The algorithm was inspired by the auditory system, in which the weighting of inputs arriving at two ears is the main cue for spatial localization. TMA reproduces tactile source positions through level weighted vibrations emitted to four actuators (prior studies show that level differences are more prominent for comparisons of somatosensory inputs than temporal cues72,82 (Frost and Richardson 1997). Weights are constructed considering the person’s head in the center and the source’s angle vector, to arrive at a difference coefficient for every actuator. This coefficient is then converted to a logarithmic scale to define the stimulus amplitude. Next, by producing gradual changes to the weighting, the algorithm enables a smooth rendition of source motion on four actuators (four fingertips) while the actuators remain static. The content to the four tactile actuators was emitted as audio via the same Dante network as the Ambisonic sound environment. The algorithm and experimental paradigm were programmed in MaxMSP.

Sound level measurements (dB levels)

Sound levels were measured in the experimental space using a MiniDSP UMIK-1 microphone (frequency response 8Hz-20kHz) placed at the participant’s head position. All measurements were done both with the tactile boxes ON and OFF. Specifically, the levels were measured for 20 randomly selected 4-s stimuli. The results were the following: a) tactile devices OFF: max. 65.3 ± 2.2dB(z) vs. b) tactile devices ON: max 64.6 ± 1.7dB(z).

Preparations

Upon arrival at the Reichman University, each participant first signed an Informed Consent, and then was briefly interviewed regarding their demographic details and hearing status. For the experimental part, the participant was seated in the center of the experimental room, with the head at the height of the center speaker ring and one tactile device on a small pedestal on the right and left side of the body (see Figures 1 and 2). The experimenter was seated in a separate control room and communicated with the participant using a talkback microphone (another microphone was placed at the ceiling of the experimental room). The experimenter could also monitor the participant using an overhead camera feed. In addition, all instructions were given to the participants in person face-to-face before each part of the experiment. The lights in the room were dimmed. Before any experiment started, it was also made sure that the participants could hear the experimental sounds inside the room, including the participants with hearing aids and cochlear implants (with their everyday device settings). All participants confirmed they could hear samples of sounds. The mobile phones of the hearing-impaired participants were placed in the control room, BlueTooth was disconnected.

Experimental procedure

The stimuli for the experiment were constructed of a sawtooth wave at a frequency of 200Hz. Both audio and tactile stimuli were designed to induce perception of motion on a horizontal plane around the person. Motion could start in any of the eight positions around the participant, i.e., front, front-right, right, back-right, back, back-left, left, front-left; see Figure 2). Motion direction could be either clockwise or counterclockwise. Each stimulus was 4 s in duration and could move at an azimuthal angle of 45°, 90°, 135°, 180°. There were four experimental conditions, in the following order for all individuals: Audio-Baseline (A1), Tactile (T), Audio-Tactile (AT) and Audio-Post (A2). For each condition there was a separate test sheet containing 28 stimuli. Test sheets were pseudo-randomly created to equally represent the various orientations in space and directions of motion. Four catch trials included stimuli which were not moving at all. Responses were given verbally (e.g.,“ The sound moved from front to right back”). Before the start of the experiment, each participant received an explanation of the localization task and was given a response sheet showing 8 possible responses (front, front-right, right, back-right, back, back-left, left, front-left). In addition, prior to the Tactile condition a brief explanation of the tactile algorithm was given and practiced with three example moving stimuli. For the Tactile condition, cochlear implant users and hearing aid users took their devices off, and the typically hearing individuals wore headphones emitting white noise (due to the noise produced by the tactile device when no concurrent sounds are present). During the Audio-Tactile condition, the participants were not able to hear the noise of the tactile devices (see level measurements above).

Quantification and statistical analysis

Task performance

The score for each condition in the localization task was calculated by combining the reported start-point, endpoint and direction to arrive at the perceived midpoint of the motion trajectory, and comparing it to the actual stimulus midpoint. The distance in angle between the perceived and the actual stimulus midpoint was then converted to a score on a scale of 0–1 (perfect response: 0° error = 1; maximum error: 180° error = 0). Since localization was performed on a circle and was calculated based on angle errors (which to either direction would result as identical in scoring), a chance error was 0.5 (translating into 90° of error). The score (in Arbitrary Units, A.U.) was then compared for every condition to chance, and between conditions (Audio-Baseline, Audio-Tactile, Tactile, Audio-Post) for the typically hearing group (N = 29), the whole hearing-impaired group (N = 19) and the bilateral cochlear implant users separately (N = 10) (Wilcoxon Signed-Rank test). The midpoint scores groups were also compared with one another, within-group, for each condition separately (Mann-Whitney tests).

Correlation analysis

A Kendall r correlation analysis was applied to examine the relationship between the results in the experiment and age at fitting of both cochlear implants in bilateral cochlear implant users.

Additional questionnaires

Nijmegen Cochlear Implant Questionnaire (NCIQ) and Abbreviated Profile of Hearing Aid Benefit (APHAB)

To assess the subjective hearing status and the related quality of life (QoL) of the hearing-impaired participants, one of the two questionnaires were used depending on the used device. Participants with one or two cochlear implants (N = 18) filled in the Nijmegen Cochlear Implant Questionnaire (NCIQ41). The original English version was translated to Hebrew by a bilingual English-Hebrew speaker. NCIQ is composed of 60 questions representing three hearing-related functional domains and (within them) 6 subdomains: 1) physical (basic sound perception, advanced sound perception, speech production), 2) psychological (self-esteem), 3) social (activity limitations, social interactions). The Basic Sound Perception scale refers to hearing sounds like the phone, car, someone calling, etc.; the Advanced Sound Perception scale refers to situations, such as having a conversation, enjoying music, recognizing gender of the speaker, etc.; Speech Production refers to the ability of controlling one’s voice pitch and volume; Self-Esteem refers to accepting one’s own deficit, building new relationships, etc; the Activity scale refers to the amount of engagement at work, in hobbies, going out, etc.; the Social Interactions scale refers to communication with different types of people and groups. The participants responded to the statements in the questionnaire as: never (1), sometimes (2), often (3), mostly (4), and always (5), or as no (1), poorly (2), moderate (3), adequate (4), and good (5) (last 5/60 items). An additional answer category was also offered, i.e., “not applicable (N/A)”. The score for a response to each item was: 1 = 0, 2 = 25, 3 = 50, 4 = 75, and 5 = 100. The total subdomain score is calculated by averaging the scores for 10 items per subdomain, with higher scores signifying better functioning. Participants who had either one or two hearing aids (N = 5), filled in the Abbreviated Profile of Hearing Aid Benefit (APHAB, version A83). APHAB is an inventory for self-assessment, composed of 24 items concerning the experienced hearing-related handicap in everyday situations. There are 4 subscales: Ease of Communication (EC), Reverberation (RV), Background Noise (BN), and Aversiveness (AV), with higher scores signifying more impaired functioning. The subjective benefit of the hearing aid is derived from comparing the results in the unaided vs. aided condition.

Color Trial Test (CTT)

In order to exclude participants with sustained and divided attention deficits, the Color Trial Test (CTT; d`Elia 1996) was applied. This standard attention test is as free as possible from the influences of language and cultural bias. Fifteen participants with hearing deficits (3 males, 12 females, age: 29.13/-11.55; 9 bilateral cochlear implant users, 1 bilateral hearing aid user, 2 with 1 hearing aid and 1 cochlear implant, 3 with 1 cochlear implant and deaf in the other ear) and 26 participants with typical hearing (7 males, 19 females, age: 23.37 ± 1.4) participated. Due to time constraints, only the more advanced CTT2 (vs. the more simple CTT1) sheet was applied, with the task to connect circles following an ascending number sequence (1–25), while alternating between circles in yellow and circles in pink. In CTT2 the same numbers are presented both on the pink and on the yellow circles, therefore requiring the participant to actively ignore some of them. The results were compared between the group of the typically hearing individuals and the hearing impaired group (Wilcoxon signed-rank tests).

Supplemental information

Document S1. Tables S1–S5

Supplemental information can be found online at https://doi.org/10.1016/j.isci.2024.110808.
==== Refs
References

1 Anderson S.R. Jocewicz R. Kan A. Zhu J. Tzeng S. Litovsky R.Y. Sound source localization patterns and bilateral cochlear implants: Age at onset of deafness effects PLoS One 17 2022 e0263516
2 Dorman M.F. Loiselle L.H. Cook S.J. Yost W.A. Gifford R.H. Sound source localization by normal-hearing listeners, hearing-impaired listeners and cochlear implant listeners Audiol. Neurootol. 21 2016 127 131 27077663
3 Moua K. Kan A. Jones H.G. Misurelli S.M. Litovsky R.Y. Auditory motion tracking ability of adults with normal hearing and with bilateral cochlear implants J. Acoust. Soc. Am. 145 2019 2498 2511 31046310
4 Shafiro V. Sheft S. Kuvadia S. Gygi B. Environmental sound training in cochlear implant users J. Speech Lang. Hear. Res. 58 2015 509 519 25633579
5 Zheng Y. Swanson J. Koehnke J. Guan J. Sound localization of listeners with normal hearing, impaired hearing, hearing aids, bone-anchored hearing instruments, and cochlear implants: a review Am. J. Audiol. 31 2022 819 834 35917460
6 Bruns P. Röder B. Development and experience-dependence of multisensory spatial processing Trends Cog. Sci. 27 2023 961
7 Spence C. Di Stefano N. What, if anything, can be considered an amodal sensory dimension? Psychon. Bull. Rev. 1 2024 1 19
8 Schnupp J. Nelken I. King A. Auditory Neuroscience: Making Sense of Sound 2011 MIT press
9 Heimler B. Amedi A. Are critical periods reversible in the adult brain? Insights on cortical specializations based on sensory deprivation studies Neurosci. Biobehav. Rev. 116 2020 494 507 32652097
10 Hubel D.H. Wiesel T.N. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex J. Physiol. 160 1962 106 154 14449617
11 Bruns P. Li L. Guerreiro M.J.S. Shareef I. Rajendran S.S. Pitchaimuthu K. Kekunnaya R. Röder B. Audiovisual spatial recalibration but not integration is shaped by early sensory experience iScience 25 2022 104439 10.1016/j.isci.2022.104439
12 Gori M. Campus C. Signorini S. Rivara E. Bremner A.J. Multisensory spatial perception in visually impaired infants Curr. Biol. 31 2021 5093 5101.e5 34555348
13 King A. Development of Multisensory Spatial Integration Crossmodal space and crossmodal attention 2004 1 24
14 Kral A. Sato M. Nature and nurture in hearing: critical periods for therapy of deafness Acoust Sci. Technol. 41 2020 54 58
15 Middlebrooks J.C. Sound localization Handb. Clin. Neurol. 129 2015 99 116 25726265
16 Carlile S. Leung J. The perception of auditory motion Trends Hear. 20 2016 2331216516644254
17 Akeroyd M.A. An overview of the major phenomena of the localization of sound sources by normal-hearing, hearing-impaired, and aided listeners Trends Hear. 18 2014 2331216514560442
18 Best V. Baumgartner R. Lavandier M. Majdak P. Kopčo N. Sound externalization: A review of recent research Trends Hear. 24 2020 2331216520948390
19 Johnson J.A. Xu J. Cox R.M. Impact of hearing aid technology on outcomes in daily life III: Localization Ear Hear. 38 2017 746 759 28700447
20 Carlyon R.P. Goehring T. Cochlear implant research and development in the twenty-first century: a critical update J. Assoc. Res. Otolaryngol. 22 2021 481 508 34432222
21 Lu L. Zhang X. Gao X. Non-implantable Artificial Hearing Technology Adv. Exp. Med. Biol. 1130 2019 145 163 30915706
22 Pavani F. Venturini M. Baruffaldi F. Artesini L. Bonfioli F. Frau G.N. van Zoest W. Spatial and non-spatial multisensory cueing in unilateral cochlear implant users Hear. Res. 344 2017 24 37 27810286
23 Coudert A. Gaveau V. Gatel J. Verdelet G. Salemme R. Farne A. Pavani F. Truy E. Truy E. Spatial hearing difficulties in reaching space in bilateral cochlear implant children improve with head movements Ear Hear. 43 2022 192 205 34225320
24 Mueller M.F. Meisenbacher K. Lai W.K. Dillier N. Sound localization with bilateral cochlear implants in noise: How much do head movements contribute to localization? Cochlear Implants Int. 15 2014 36 42 23684420
25 Nisha K.V. Uppunda A.K. Kumar R.T. Spatial rehabilitation using virtual auditory space training paradigm in individuals with sensorineural hearing impairment Front. Neurosci. 16 2022 1080398
26 Dwyer R.T. Chen C. Hehrmann P. Dwyer N.C. Gifford R.H. Synchronized automatic gain control in bilateral cochlear implant recipients yields significant benefit in static and dynamic listening conditions Trends Hear. 25 2021 23312165211014139
27 Fischer T. Schmid C. Kompis M. Mantokoudis G. Caversaccio M. Wimmer W. Pinna-imitating microphone directionality improves sound localization and discrimination in bilateral cochlear implant users Ear Hear. 42 2021 214 222 32701730
28 Snir A. Ciesla K. Vekslar R. Ozdemir G. Amedi A. Localizing 3D Motion through Fingertips: Following in the Footsteps of Elephants iScience 27 2024 109820
29 O’Connell-Rodwell C.E. Keeping an “ear” to the ground: seismic communication in elephants Physiology 22 2007 287 17699882
30 Kalmijn A.J. Functional Evolution of Lateral Line and Inner Ear Sensory Systems The mechanosensory lateral line: neurobiology and evolution 1989 Springer New York 187 215
31 Beauchamp M.S. Yasar N.E. Frye R.E. Ro T. Touch, sound and vision in human superior temporal sulcus Neuroimage 41 2008 1011 1020 18440831
32 Caetano G. Jousmäki V. Evidence of vibrotactile input to human auditory cortex Neuroimage 29 2006 15 28 16168673
33 Kayser C. Petkov C.I. Augath M. Logothetis N.K. Integration of touch and sound in the auditory cortex Neuron 48 2005 373 384 16242415
34 Li Hegner Y. Lee Y. Grodd W. Braun C. Comparing tactile pattern and vibrotactile frequency discrimination: a human FMRI study J. Neurophysiol. 103 2010 3115 3122 20457848
35 Ro T. Ellmore T.M. Beauchamp M.S. A neural link between feeling and hearing Cereb. Cortex 23 2013 1724 1730 22693344
36 v Békésy G. Sensations on the skin similar to directional hearing, beats, and harmonics of the ear J. Acoust. Soc. Am. 29 1957 489 501
37 Yu H.V. Tao L. Llamas J. Wang X. Nguyen J.D. Trecek T. Segil N. POU4F3 pioneer activity enables ATOH1 to drive diverse mechanoreceptor differentiation through a feed-forward epigenetic mechanism Proc. Natl. Acad. Sci. USA 118 2021 e2105137118
38 Heimler B. Striem-Amit E. Amedi A. Origins of task-specific sensory-independent organization in the visual and auditory brain: neuroscience evidence, open questions and clinical implications Curr. Opin. Neurobiol. 35 2015 169 177 26469211
39 Bach-y-Rita P. Collins C.C. Saunders F.A. White B. Scadden L. Vision substitution by tactile image projection Nature 221 1969 963 964 5818337
40 D'Elia L. Satz P. Uchiyama C.L. White T. Color Trails Test 1996 PAR
41 Hinderink J.B. Krabbe P.F. Van Den Broek P. Development and application of a health-related quality-of-life instrument for adults with cochlear implants: the Nijmegen cochlear implant questionnaire Otolaryngol. Head Neck Surg. 123 2000 756 765 11112975
42 Rasmussen K.M.B. West N.C. Bille M. Sandvej M.G. Cayé-Thomasen P. Cochlear implantation improves both speech perception and patient-reported outcomes: a prospective follow-up study of treatment benefits among adult cochlear implant recipients J. Clin. Med. 11 2022 2257 35456353
43 Santos N.P.D. Couto M.I.V. Martinho-Carvalho A.C. Nijmegen Cochlear Implant Questionnaire (NCIQ): Translation, Cultural Adaptation, and Application in Adults with Cochlear Implants Codas 29 2017 Sociedade Brasileira de Fonoaudiologia e20170007
44 Andersson, J. (2015). Horizontal Sound Localization in Adults with Unilateral or Bilateral Cochlear Implants.
45 Ludwig A.A. Meuret S. Battmer R.D. Schönwiesner M. Fuchs M. Ernst A. Sound localization in single-sided deaf participants provided with a cochlear implant Front. Psychol. 12 2021 753339
46 Lundbeck M. Grimm G. Hohmann V. Laugesen S. Neher T. Sensitivity to angular and radial source movements as a function of acoustic complexity in normal and impaired hearing Trends Hear. 21 2017 2331216517717152
47 Kan A. Litovsky R.Y. Binaural hearing with electrical stimulation Hear. Res. 322 2015 127 137 25193553
48 Laback B. Egger K. Majdak P. Perception and coding of interaural time differences with bilateral cochlear implants Hear. Res. 322 2015 138 150 25456088
49 Warnecke M. Peng Z.E. Litovsky R.Y. The impact of temporal fine structure and signal envelope on auditory motion perception PLoS One 15 2020 e0238125
50 Schäfer E. Vedoveli A.E. Righetti G. Gamerdinger P. Knipper M. Tropitzsch A. Karnath H.O. Braun C. Li Hegner Y. Li Hegner Y. Activities of the right temporo-parieto-occipital junction reflect spatial hearing ability in cochlear implant users Front. Neurosci. 15 2021 613101
51 Gordon K.A. Wong D.D.E. Papsin B.C. Bilateral input protects the cortex from unilaterally-driven reorganization in children who are deaf Brain 136 2013 1609 1625 23576127
52 Gori M. Multisensory integration and calibration in children and adults with and without sensory and motor disabilities Multisens. Res. 28 2015 71 99 26152053
53 Abboud S. Hanassy S. Levy-Tzedek S. Maidenbaum S. Amedi A. EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution Restor. Neurol. Neurosci. 32 2014 247 257 24398719
54 Kupers R. Ptito M. Cross-modal brain plasticity in congenital blindness: lessons from the tongue display unit. i-Perception Iperception. 2 2011 748
55 Maidenbaum S. Hanassy S. Abboud S. Buchs G. Chebat D.R. Levy-Tzedek S. Amedi A. The “EyeCane” Restor. Neurol. Neurosci. 32 2014 813 824 25201814
56 Meijer P.B. An experimental system for auditory image representations IEEE Trans. Biomed. Eng. 39 1992 112 121 1612614
57 Battal, C. (2018). Decoding Auditory Motion Direction and Location in hMT+/V5 and Planum Temporale of Sighted and Blind Individuals.
58 Rezk M. Cattoir S. Battal C. Occelli V. Mattioni S. Collignon O. Shared representation of visual and auditory motion directions in the human middle-temporal cortex Curr. Biol. 30 2020 2289 2299.e8 32442465
59 Arbel R. Heimler B. Amedi A. Face shape processing via visual-to-auditory sensory substitution activates regions within the face processing networks in the absence of visual experience Front. Neurosci. 16 2022 921321
60 Odegaard B. Wozny D.R. Shams L. A simple and efficient method to enhance audiovisual binding tendencies PeerJ 5 2017 e3143 10.7717/peerj.3143
61 Hecht D. Reiner M. Sensory dominance in combinations of audio, visual and haptic stimuli Exp. Brain Res. 193 2009 307 314 18985327
62 Tranchant P. Shiell M.M. Giordano M. Nadeau A. Peretz I. Zatorre R.J. Feeling the beat: Bouncing synchronization to vibrotactile music in hearing and early deaf people Front. Neurosci. 11 2017 507
63 Landry S.P. Guillemot J.P. Champoux F. Temporary deafness can impair multisensory integration: a study of cochlear-implant users Psychol. Sci. 24 2013 1260 1268 23722977
64 Landry S.P. Guillemot J.P. Champoux F. Audiotactile interaction can change over time in cochlear implant users Front. Hum. Neurosci. 8 2014 316 24904359
65 Kamke M.R. Van Luyn J. Constantinescu G. Harris J. Contingent capture of involuntary visual spatial attention does not differ between normally hearing children and proficient cochlear implant users Restor. Neurol. Neurosci. 32 2014 799 811 25201816
66 Valzolgher C. Gatel J. Bouzaid S. Grenouillet S. Todeschini M. Verdelet G. Salemme R. Gaveau V. Truy E. Farnè A. Pavani F. Reaching to sounds improves spatial hearing in bilateral cochlear implant users Ear Hear. 44 2023 189 198 35982520
67 Nava E. Bottari D. Villwock A. Fengler I. Büchner A. Lenarz T. Röder B. Audio-tactile integration in congenitally and late deaf cochlear implant users PLoS One 9 2014 e99606
68 Carlile S. Leung J. The perception of auditory motion Trends Hear. 20 2016 2331216516644254
69 Weisenberger J.M. Percy M.E. The transmission of phoneme-level information by multichannel tactile speech perception aids Ear Hear. 16 1995 392 406 8549895
70 Fletcher M.D. Hadeedi A. Goehring T. Mills S.R. Electro-haptic enhancement of speech-in-noise performance in cochlear implant users Sci.c Rep. 9 2019 11428
71 Cieśla K. Wolak T. Lorens A. Mentzel M. Skarżyński H. Amedi A. Effects of training and using an audio-tactile sensory substitution device on speech-in-noise understanding Sci. Rep. 12 2022 1 16 34992227
72 Fletcher M.D. Cunningham R.O. Mills S.R. Electro-haptic enhancement of spatial hearing in cochlear implant users Sci. Rep. 10 2020 1621 32005889
73 Gori M. Vercillo T. Sandini G. Burr D. Tactile feedback improves auditory spatial localization Front. Psychol. 5 2014 1121
74 Branje C. Maksimouski M. Karam M. Fels D.I. Russo F. Vibrotactile Display of Music on the Human Back 2010 Third International Conference on Advances in Computer-Human Interactions 2010 IEEE 154 159
75 Hartcher-O’Brien J. Auvray M. The process of distal attribution illuminated through studies of sensory substitution Multisens. Res. 27 2014 421 441 25693304
76 Asp F. Karltorp E. Berninger E. Development of Sound Localization in Infants and Young Children with Cochlear Implants J. Clin. Med. 11 2022 6758 36431235
77 Moore D.R. Auditory development and the role of experience Br. Med. Bull. 63 2002 171 181 12324392
78 Alzaher M. Valzolgher C. Verdelet G. Pavani F. Farnè A. Barone P. Marx M. Audiovisual training in virtual reality improves auditory spatial adaptation in unilateral hearing loss patients J. Clin. Med. 12 2023 2357 36983357
79 Yizhar O. Buchs G. Heimler B. Friedman D. Amedi A. Applying a novel visual-to-touch sensory substitution for studying tactile reference frames Sci. Rep. 11 2021 10636
80 Badde S. Heed T. Towards explaining spatial touch perception: Weighted integration of multiple location codes Cogn. Neuropsychol. 33 2016 26 47 27327353
81 Cieśla K. Wolak T. Lorens A. Heimler B. Skarżyński H. Amedi A. Immediate improvement of speech-in-noise perception through multisensory stimulation via an auditory to tactile sensory substitution Restor. Neurol. Neurosci. 37 2019 155 166 31006700
82 Borg E. Cutaneous senses for detection and localization of environmental sound sources: a review and tutorial Scand. Audiol. 26 1997 195 206 9428027
83 Cox R.M. Alexander G.C. The abbreviated profile of hearing aid benefit Ear Hear. 16 1995 176 186 7789669
