==== Front PLoS One PLoS One plos PLOS ONE 1932-6203 Public Library of Science San Francisco, CA USA 10.1371/journal.pone.0287900 PONE-D-22-27111 Research Article Biology and Life Sciences Neuroscience Cognitive Science Cognitive Psychology Perception Sensory Perception Vision Biology and Life Sciences Psychology Cognitive Psychology Perception Sensory Perception Vision Social Sciences Psychology Cognitive Psychology Perception Sensory Perception Vision Biology and Life Sciences Neuroscience Sensory Perception Vision Biology and Life Sciences Neuroscience Cognitive Science Cognitive Psychology Perception Sensory Perception Sensory Cues Biology and Life Sciences Psychology Cognitive Psychology Perception Sensory Perception Sensory Cues Social Sciences Psychology Cognitive Psychology Perception Sensory Perception Sensory Cues Biology and Life Sciences Neuroscience Sensory Perception Sensory Cues Biology and Life Sciences Neuroscience Cognitive Science Cognitive Psychology Perception Sensory Perception Biology and Life Sciences Psychology Cognitive Psychology Perception Sensory Perception Social Sciences Psychology Cognitive Psychology Perception Sensory Perception Biology and Life Sciences Neuroscience Sensory Perception Research and Analysis Methods Bioassays and Physiological Analysis Electrophysiological Techniques Brain Electrophysiology Electroencephalography Event-Related Potentials Biology and Life Sciences Physiology Electrophysiology Neurophysiology Brain Electrophysiology Electroencephalography Event-Related Potentials Biology and Life Sciences Neuroscience Neurophysiology Brain Electrophysiology Electroencephalography Event-Related Potentials Biology and Life Sciences Neuroscience Brain Mapping Electroencephalography Event-Related Potentials Medicine and Health Sciences Clinical Medicine Clinical Neurophysiology Electroencephalography Event-Related Potentials Research and Analysis Methods Imaging Techniques Neuroimaging Electroencephalography Event-Related Potentials Biology and Life Sciences Neuroscience Neuroimaging Electroencephalography Event-Related Potentials Biology and Life Sciences Psychology Behavior Behavioral Conditioning Social Sciences Psychology Behavior Behavioral Conditioning Biology and Life Sciences Neuroscience Cognitive Science Cognitive Psychology Learning Biology and Life Sciences Psychology Cognitive Psychology Learning Social Sciences Psychology Cognitive Psychology Learning Biology and Life Sciences Neuroscience Learning and Memory Learning Biology and Life Sciences Neuroscience Cognitive Science Cognitive Psychology Attention Biology and Life Sciences Psychology Cognitive Psychology Attention Social Sciences Psychology Cognitive Psychology Attention Biology and Life Sciences Psychology Behavior Conditioned Response Social Sciences Psychology Behavior Conditioned Response Differential effects of intra-modal and cross-modal reward value on perception: ERP evidence Intra-modal and cross-modal effects of reward on perception Vakhrushev Roman Conceptualization Data curation Formal analysis Investigation Methodology Software Validation Visualization Writing – original draft Writing – review & editing 1 Cheng Felicia Pei-Hsin Formal analysis Methodology Writing – review & editing 1 Schacht Anne Methodology Validation Writing – review & editing 2 https://orcid.org/0000-0002-4369-8838 Pooresmaeili Arezoo Conceptualization Formal analysis Funding acquisition Investigation Methodology Project administration Resources Supervision Validation Writing – original draft Writing – review & editing 1 * 1 Perception and Cognition Lab, European Neuroscience Institute Goettingen- A Joint Initiative of the University Medical Center Goettingen and the Max-Planck-Society, Goettingen, Germany 2 Affective Neuroscience and Psychophysiology Laboratory, Georg-Elias-Müller-Institute of Psychology, Georg-August University, Goettingen, Germany Megna Nicola Editor Istituto di Ricerca e di Studi in Ottica e Optometria, ITALY Competing Interests: The authors declare no competing interests. * E-mail: a.pooresmaeili@eni-g.de 30 6 2023 2023 18 6 e028790030 9 2022 15 6 2023 © 2023 Vakhrushev et al 2023 Vakhrushev et al https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. In natural environments objects comprise multiple features from the same or different sensory modalities but it is not known how perception of an object is affected by the value associations of its constituent parts. The present study compares intra- and cross-modal value-driven effects on behavioral and electrophysiological correlates of perception. Human participants first learned the reward associations of visual and auditory cues. Subsequently, they performed a visual discrimination task in the presence of previously rewarded, task-irrelevant visual or auditory cues (intra- and cross-modal cues, respectively). During the conditioning phase, when reward associations were learned and reward cues were the target of the task, high value stimuli of both modalities enhanced the electrophysiological correlates of sensory processing in posterior electrodes. During the post-conditioning phase, when reward delivery was halted and previously rewarded stimuli were task-irrelevant, cross-modal value significantly enhanced the behavioral measures of visual sensitivity, whereas intra-modal value produced only an insignificant decrement. Analysis of the simultaneously recorded event-related potentials (ERPs) of posterior electrodes revealed similar findings. We found an early (90–120 ms) suppression of ERPs evoked by high-value, intra-modal stimuli. Cross-modal stimuli led to a later value-driven modulation, with an enhancement of response positivity for high- compared to low-value stimuli starting at the N1 window (180–250 ms) and extending to the P3 (300–600 ms) responses. These results indicate that sensory processing of a compound stimulus comprising a visual target and task-irrelevant visual or auditory cues is modulated by the reward value of both sensory modalities, but such modulations rely on distinct underlying mechanisms. http://dx.doi.org/10.13039/501100000781 European Research Council 716846 https://orcid.org/0000-0002-4369-8838 Pooresmaeili Arezoo This work was supported by an ERC Starting Grant (no: 716846) to AP. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data AvailabilityThe data and analysis scripts that support the findings of this study are uploaded to the Open Science Framework data repository (https://osf.io/47wxr/files/osfstorage#). Further information regarding the data and the analysis scripts can be obtained from the corresponding author (AP). Data Availability The data and analysis scripts that support the findings of this study are uploaded to the Open Science Framework data repository (https://osf.io/47wxr/files/osfstorage#). Further information regarding the data and the analysis scripts can be obtained from the corresponding author (AP). ==== Body pmcIntroduction Reward seeking is a fundamental mechanism for survival and one of the main predictors of behavior [1, 2]. We tend to prioritize what we eat, where we go and what we do based on the expected value of objects or actions learned through experience. A large body of literature has identified a network encompassing the ventral striatum and orbitofrontal cortex to play a key role in learning the associated value of neutral stimuli through experience [3, 4]. As natural environments are rich and dynamic, reward information could be conveyed through multiple sensory modalities (e.g., hearing the sound of an approaching ice-cream truck or seeing its characteristic red color both inform us of the possibility of enjoying an ice cream) and reward associations or the goals of the task at hand may change over time (e.g., a red truck can carry other items rather than ice-cream). These features entail a tight interaction between the reward network and the early sensory areas so that stimuli leading to better outcomes such as higher rewards or realization of the goals of the task are prioritized for perceptual processing [5, 6]. In fact, previous research has identified value-driven modulations of neuronal responses in almost all primary sensory areas [7–12]. In line with the effect of reward on the earliest stages of sensory processing, studies on humans with the use of electroencephalography (EEG) reported reward-related modulations of the visual event-related potentials (ERPs) that are likely to originate from the primary and extrastriate visual cortices [but see also 13], including modulations in P1 (80–120 ms) [14–16], N1 (140–220 ms) [17] and C1 (~70 ms) [18, 19] ERP components. Despite the robust evidence for an influence of reward on early sensory mechanisms, especially in the visual cortex, it has remained unclear how reward information and sensory processing are coordinated across multiple sensory modalities. This is an important question as in natural environments stimuli are typically multisensory and reward information should be hence coordinated across multiple sensory modalities. Two recent studies tried to bridge this gap and examined cross-modal reward mechanisms, where an auditory stimulus associated with positive reward value affects visual perception [20, 21]. During a conditioning phase, participants associated different pure tones with different reward values. In a subsequent post-conditioning phase, participants either reported the location [20] or the orientation [21] of a near-threshold Gabor stimulus in the presence of task-irrelevant auditory tones. Importantly, during the post-conditioning phase, auditory tones did not predict the delivery of reward anymore. The first study [20] showed that participants’ accuracy in determining the location of the visual target was higher in the presence of tones previously associated with a positive reward compared to no reward (neutral). The authors concluded that the tones associated with rewards enhanced the bottom-up salience of the visual stimuli. Similar results were found in the second study [21], as it was shown that participants’ orientation discrimination improved in the presence of auditory tones previously associated with high compared to low reward value. With the aid of the simultaneously acquired functional MRI data, it was further shown that the enhanced orientation discrimination is also reflected in the activation patterns of the early visual areas elicited by a specific stimulus tilt orientation [21]. Moreover, in addition to the classical reward coding areas (such as Striatum and Orbitofrontal cortex), multisensory regions of the temporal cortex were also modulated by sound values suggesting that they may serve as an intermediary stage to better coordinate the interaction between the primary sensory cortices when high-value stimuli were presented [21]. Subsequent studies [22–25] used a variety of tasks encompassing visual search or detection or discrimination tasks and confirmed that cross-modal (auditory) reward cues could affect visual perception, albeit the cross-modal reward cues in some cases interfered with the visual task [22, 25] and in other cases improved it [23, 24]. Together, these results provide evidence that reward effects can occur cross-modally. However, it remains unclear whether the modulatory effects of reward on perception depend on whether reward cues are from the same or different sensory modality as the task-relevant target stimulus. A predominant view posits that reward effects on sensory perception occur through the engagement of attentional mechanisms [26–29]. In this view, rewarding cues receive higher attentional prioritization either through an involuntary, value-driven attentional capture [30], through voluntary, goal-directed attentional selection [31–34], or through cognitive control mechanisms [35, 36]. In line with this, it has been shown that a reward cue that is aligned with the goal-directed attention in space and in time improves visual performance [31, 37], whereas when the rewarded stimulus is presented away from the target position, it interferes with the task [38, 39]. Considering these findings, enhancement of visual perception by co-occurring sounds [20, 21] is unexpected, as auditory tones in these studies were irrelevant to the visual task and could potentially act as a high-reward distractor capturing attention away from the visual target. One possibility is that cross-modal reward cues enhance visual perception by strengthening the audiovisual integration of the auditory and visual components of an audiovisual stimulus. Although audiovisual integration largely occurs automatically [for a review see 40], top-down factors such as attention [41–43] and recently reward value [44, 45] have been shown to affect its strength. Through a more efficient integration with the visual cues, auditory reward signals could hence capture attention not only to themselves but also to the whole audiovisual object including the visual target [46], thereby improving performance. In fact, the boost of integration may be a key characteristic of reward modulation where the association of one sensory property of an object with higher reward spreads to all sensory properties of the same object hence promoting their grouping, as has been shown before in the visual modality [11]. Therefore, although previous research has provided evidence for a spread of reward-driven effects across different parts of visual [11] and audiovisual objects [20, 21], these effects have never been compared against each other. The central aim of this study was to provide a better understanding of reward effects across sensory modalities by comparing intra- and cross-modal reward effects, while using the high temporal resolution of the EEG data to delineate the different stages of stimulus processing across time. To this end, we employed a task design similar to a previous study [21], where participants first learned the reward value of visual or auditory cues during a conditioning phase and subsequently performed a visual orientation task in the presence of previously rewarded cues. During the latter post-conditioning phase, auditory and visual reward cues were presented simultaneously with the target stimulus and at the same side of the visual field. Simultaneous presentation of reward cues and visual target allowed us to test whether the reward associations of task-irrelevant parts of a compound object (i.e. previously rewarded auditory and visual cues) can affect the processing of the task-relevant visual target. In this case, the spread of value-driven modulations from the visual or auditory cues to the target will indicate object-based effects in the visual modality [47] or cross-modally [46]. However, since this design compared visual and audiovisual stimuli that are known to elicit different magnitudes of neural responses [48], intra-modal and cross-modal high reward conditions were contrasted with their low reward counterpart from the same sensory configuration, thus allowing us to isolate the reward-driven effects independent of the sensory configuration of stimuli. Crucially, reward cues during the post-conditioning phase were irrelevant to the task at hand and did not predict the delivery of reward (i.e. no-reward phase). Measuring reward effects at a no-reward phase has proven to be an effective method to separate different modulations of perception, i.e., those related to the long-term associative value of reward cues from those of goal-directed boosts driven by reward-predicting cues or the delivery of the reward itself [49–54]. We hypothesized that cues previously associated with high value should positively influence behavioral performance (i.e., enhancing target’s discriminability and reducing the reaction times) and early posterior ERP components (i.e., increasing the amplitudes of P1 and N1 components, and decreasing their latencies). This hypothesis is based on a mechanism in which value-driven prioritization of task-irrelevant reward-associated cues spreads to other sensory components of the same object and thereby enhances the representation of the target irrespective of the sensory modality of the reward cue. Next, we hypothesized that reward-related modulations of P1 and N1 components in posterior electrodes occur earlier and are stronger when the reward cue is delivered intra-modally (in visual modality) compared to when the reward cue is delivered cross-modally (in auditory modality), as intra-modal effects rely on direct neuronal connections between reward and target sensory representations [55, 56], whereas cross-modal effects rely on the long-range communication between different brain areas and/or involve intermediate stages [21]. Finally, we hypothesized that in later stages of information processing (i.e. > 250 ms), intra- and cross-modal reward cues elicit similar modulations of ERP amplitudes, both leading to an enhancement of P3 ERP component. Material and methods Participants Thirty-eight participants took part in our experiment. Two participants were excluded since their performance during the associative reward learning task indicated that they either did not learn the reward associations (N = 1) or had difficulties with discriminating the location of the stimuli (N = 1). The final sample consisted of 36 healthy participants (23 women, mean age ± SD: 25.8 ± 5 years; 28 right-handed) with normal or corrected-to-normal vision who had no history of neurophysiological or psychiatric disorders according to a self-report. Participants gave written informed consent, after the experimental procedures were clearly explained to them. The study was conducted in full accordance with the Declaration of Helsinki and was approved by the local Ethics Committee of the Medical University Göttingen (proposal: 15/7/15). The sample size, all procedures, and the analysis plan of the study were preregistered (https://osf.io/47wxr/). The required sample size was calculated based on a pilot study (N = 8) indicating that at least 33 subjects were needed to detect a significant difference between high and low reward value (α = 0.05, 1 - β = 0.8; GPower [57]). To maintain the counterbalancing of our experimental conditions (4 possible combinations of auditory and visual cues with high or low reward), we aimed for an a priori sample size of N = 36 before the data collection started. Experimental procedures Data collection was done in a darkened, sound-attenuated, and electromagnetically shielded chamber. Participants sat 91 cm away from a 22.5-inch calibrated monitor (ViewPixx/EEG inc., resolution = 1440 × 980 pixels, refresh rate = 120 Hz) with their heads resting on a chinrest. Stimulus presentation was controlled in Psychophysics toolbox-3 [58] in MATLAB (version R2015b) environment. Eye movements were recorded with an Eyelink 1000 eye tracker system (SR Research, Ontario, Canada) in a desktop mount configuration, recording the right eye at a sampling rate of 1000 Hz. Electrophysiological data were recorded from 64 electrodes (BrainVision Recorder 1.23.0001 Brain Products GmbH, Gilching, Germany; actiCap, Brain Products GmbH, Gilching, Germany), online referenced to TP9, digitized at 1000 Hz, and amplified with a gain of 10,000. Electrode impedances were kept below 10kΩ. Each participant performed two tasks: an associative reward learning task (conditioning phase) (Fig 1A) and a visual orientation discrimination task in the presence of task-irrelevant auditory or visual cues (pre- and post-conditioning phases, Fig 1B). An experimental session started with a calibration procedure, where for each participant the luminance of two consecutively presented colors was adjusted until the perceived flicker between them was minimized and they became perceptually isoluminant (total duration of calibration < 5min). The calibration was followed by a short training session for the orientation discrimination task (Number of trials = 36). After training, each participant’s orientation discrimination threshold was determined using the QUEST method (an adaptive psychometric procedure that selects the stimulus intensity- here the tilt orientation- on each trial at the current most probable Bayesian estimate of threshold [59]). This procedure was adjusted to find a tilt degree where a participant reached an accuracy level of 70% as their perceptual threshold. Thereafter the session proceeded to the pre-conditioning, conditioning and post-conditioning phases (Fig 1C). 10.1371/journal.pone.0287900.g001 Fig 1 Stimuli and experimental procedures. a) Reward learning phase (conditioning). Participants learned the reward association of the visual or auditory cues by performing a localization task and observing their monetary rewards contingent on the color (visual) or pitch (auditory) of the cues. In this task, after an initial fixation period (700–1400 ms), a visual or an auditory cue was presented to the left or right side of the fixation point and participants had to localize them by pressing either left or right arrow buttons of a keyboard (maximum response time 2 s). Here, stimuli in two example trials are shown, one with an auditory cue presented to the left and the other with a visual cue presented to the right. The correct performance led to either a high or a low monetary reward dependent on the identity of the cue (color of visual cues and pitch of auditory cues). b) Orientation discrimination task employed during the pre- and post-conditioning phases. To probe the effects of reward value on visual sensitivity, an orientation discrimination task was employed. A trial of this task started with a fixation period (700–1400 ms), followed by the presentation of a peripheral Gabor stimulus (9° eccentricity). Participants were instructed to discriminate the orientation of the Gabor stimulus (clockwise or counterclockwise tilt) by pressing down or up arrow buttons on a keyboard (maximum response duration = 2 s). Concurrent with the Gabor, a visual or auditory cue was also presented (intra- or cross-modal cues, respectively). Intra- and cross-modal cues were irrelevant to the orientation discrimination task and did not predict reward delivery. c) The schematic illustration of the different stages of an experimental session. Note that after a long conditioning block where participants learned the stimulus-reward associations, smaller blocks of conditioning were interleaved with the blocks of post-conditioning (16 repetitions) to prevent the extinction of reward effects. d) Design matrix of stimulus conditions used during the pre- and post-conditioning phases to assess the effect of reward (high or low) and sensory modality (intra- or cross-modal). A neutral condition was also included during the pre- and post-conditioning phases, which was never associated with any reward value and served to assess the responses evoked by the visual target. Note that the reward assignments of visual and auditory cues were counterbalanced across participants. During pre- and post-conditioning phases, participants were instructed to report the tilt orientation of a Gabor stimulus relative to the horizontal meridian by pressing a keyboard button (down and up arrow keys for clockwise and counterclockwise tilts, respectively). Gabor stimuli were Gaussian-windowed sinusoidal gratings with an SD of 0.33°, a spatial frequency of 3 cycles per degree, 2° diameter, and 50% contrast, displayed on a gray background of 34.4 cd/m2 luminance, presented at 9° eccentricity. A transparent ring (0.44° in diameter, 0.17 pixels thick, alpha 50%) was overlaid on the Gabor in all trials. The color of the ring depended on the experimental condition (Fig 1D): blue or orange colors were paired with different reward magnitudes during the conditioning phase and are referred to as intra-modal reward cues, whereas the grey color was never paired with rewards and was used in cross-modal and neutral conditions (color in HSV = [20, 0.4, V] for orange, [250, 0.4, V] for blue, and [0, 0, V] for grey, where V was calibrated for each participant to produce isoluminance). Auditory cues were pure monaural tones, lateralized to the left or right earphone (sound level pressure = 70 dB, delivered through earphones: ER-3C, Etymotic Research Inc.) and were presented simultaneously with the Gabor target. In cross-modal conditions, different tone frequencies (350 Hz or 1050 Hz) were associated with either high or low reward during the conditioning. Therefore, in our design only one feature of a stimulus, either the color of the ring in intra-modal stimuli or the pitch of the tones in cross-modal stimuli, had a distinct pairing with a high or low reward, whereas other features (a Gabor and an overlaid circle) were shared among all stimuli. Participants were instructed to focus on judging the orientation of the Gabor and ignore the task-irrelevant visual and auditory cues, and were additionally told that correct responses in the orientation discrimination task granted them a fixed reward shown at the end of the experiment. These instructions aimed to emphasize that visual and auditory cues were not relevant to the task and did not lead to differential immediate rewards after conditioning. In total, participants performed 160 trials in the pre-conditioning phase and 320 trials in the post-conditioning phase, with 32 and 64 repetitions of each of the value (high, low) × modality (intra-modal, cross-modal) conditions, as well as the neutral condition. During the conditioning phase (Fig 1A), participants reported the location of a peripheral visual (9° eccentricity) or auditory stimulus (sound played in the left or right earphone) by pressing left or right arrow keyboard buttons. Visual and auditory cues had the same characteristics as in pre- and post-conditioning. However, note that the neutral condition included during the orientation discrimination task, was not presented to the participants during the conditioning and was therefore not associated with any reward value. Correct responses were followed by a monetary reward, the magnitude of which depended on the respective color or pitch of the visual or auditory cues (in case of an error the monetary reward was set to zero). Reward information was presented at the screen center both in written format (with digits showing the number of Euro cents obtained) and graphically (a color bar illustrating the ratio of the reward magnitude in current trial to the maximum possible level in the experiment, respectively), and stayed in view for 1 s. Rewards were drawn from two Poisson distributions with the mean of 25 cents and SD of 4.8 cents for high-reward trials and a mean of 2 cents and SD 1.4 cents for low-reward trials (minimum and maximum reward was fixed at 35 and 0.4 cents, respectively). Participants completed 280 trials of the cue localization task to learn the reward associations. From these, 144 trials were presented during an initial longer conditioning block (comprising 36 repetitions of value × modality conditions). Subsequently, 136 trials of the conditioning phase were divided into 16 short blocks (8 trials per block) and then interleaved with short blocks of the orientation discrimination task during post-conditioning (20 trials per block) to prevent the extinction of the reward associations (Fig 1C). Participants were instructed to remember and report the color and the sound pitch that delivered higher rewards at the end of the conditioning phase and at the end of the experiment (by indicating which of the two sequentially presented colored circles or auditory tones had higher rewards). Throughout the experiment, trials were repeated in case participants made an eye movement (> 2° displacement of gaze from the fixation dot during fixation or target period), used an undesignated button to respond, or did not respond at all to maintain equal number of repetitions across conditions. The location (left or right), modality (auditory or visual), and identity (color or sound pitch) of the cues were pseudorandomized across trials with the constraint that the same condition could not be repeated more than twice in a row. The association of each cue identity with the reward value was counterbalanced across participants. This means that each reward condition (high or low), comprised an equal number of instances where either the orange or the blue color or the 350 Hz or the 1050 Hz tone was associated with that reward magnitude. Therefore, when describing the effects during the orientation discrimination task, we will refer to all stimulus conditions with respect to the reward assignment that they acquired after the conditioning, although during the pre-conditioning these associations were not learned yet. Accordingly, these conditions in both pre- and post-conditioning will be referred to as: Intra-modal High Reward: IH, Intra-modal Low Reward: IL, Cross-modal High Reward: CH and Cross-modal Low Reward: CL, plus a neutral condition (referred to as Neut), which was never associated with any reward value (as shown in Fig 1). Also note that the neutral condition comprised a Gabor patch and a semi-transparent ring overlaid on it, a feature that was shared among all stimulus configurations. Since this condition was never associated with any reward, it served as a means to characterize the behavioral and neural responses to the visual target, independently from the responses to reward-associated features of the stimuli (i.e. colors and sound pitches). Data analysis The main focus of our study was on the value-driven effects on behavior and ERPs. However, following our pre-registered plan, we also examined the effect of rewards on the pupil size and measured the correlation between reward effects and participants’ scores on a standard reward sensitivity test and these results are reported in S1 Text. Analysis of the behavioral data For behavioral assessment of visual sensitivity during the orientation discrimination task, we used participants’ d-prime scores (d’). D-prime was measured based on the probability of hits and false-alarms, as d’ = Z(PHit)—Z(PFA), where one of the tilt directions was arbitrarily treated as “target-present” as in formal Signal Detection Theory analysis of discrimination tasks [60, 61]. Extreme values of PHit or PFA were slightly up- or down-adjusted (i.e., a probability equal to 0 or 1 was adjusted by adding or subtracting 0.001, respectively). Reaction times in all phases of the experiment were calculated as the elapsed time between the onset of the target stimulus and the participant’s response. The resulting response times were averaged for each experimental condition (including correct and incorrect responses). Outliers were removed from the behavioral and ERP data of each participant (0.38% ± 0.6 SD of all trials across all subjects). A trial was considered to be an outlier if the gaze fixation during the target presentation period was suboptimal (eye position >0.9° from the fixation point) or the response buttons had been pressed prior to the presentation of the target (i.e., reaction times <10 ms). Analysis of event-related potentials (ERPs) The EEG data was imported and processed offline using EEGLAB [62], an open-source toolbox running under the MATLAB environment. First, an automatic bad channel detection and removal algorithm was applied (using EEGLAB’s pop_rejchan method; threshold = 5, method = kurtosis). Later, data of each participant was band-pass filtered with 0.1 Hz as the high-pass cutoff and 40 Hz as the low-pass cutoff frequencies. After this, the epochs were extracted by using a stimulus-locked window of 1900 ms (-700 to 1200 ms) and subjected to an independent component analysis (ICA) algorithm [62]. Blinks and eye-movement artifacts were automatically identified and corrected using an ICA-based automatic method, implemented in the ADJUST plugin of EEGLAB [63]. Bad channels were interpolated using the default spherical interpolation method of the EEGLAB toolbox. Next, data were re-referenced to the average reference. To calculate ERPs, shorter epochs of 900 ms were extracted between -100 ms to 800 ms relative to the onset of the target, and baseline-corrected using the pre-stimulus time interval (-100 to 0 ms). Our pre-registered ERP analysis focused mainly on the latencies and amplitudes of P1 and N1 components of the stimulus-evoked activity in occipital and parietal visual areas. To this end, ERPs were averaged across a region of interest (ROI) comprising four posterior electrodes (PO7/PO8, O1/O2). The peaks of P1 and N1 components were defined as the most positive (P1) or the most negative (N1) deflections of the grand-average ERP waveforms occurring 70–170 ms or 180–250 ms after the target onset, respectively. P1 and N1 amplitudes were calculated as the mean activity in a 30 ms window centered on each component’s respective peak and the timing of the peak defined each component’s peak latency. To determine the onset latency of intra- and cross-modal reward modulations, we calculated the difference between high- and low-value ERPs of each cue type within a 10 ms moving window against their baseline difference—within 100 ms before the stimulus onset- [64, 65]. Amplitude differences between 50–800 ms after stimulus onset that reached significance (uncorrected p < 0.05) in two or more consecutive time windows are reported [66]. Additionally, we planned to measure the amplitude of P3 component in midline electrodes (Fz, FCz, Cz, CPz, and Pz; 300–600 ms). After data acquisition and visual inspection of the ERPs, we undertook the following exploratory analyses. Firstly, we performed an exploratory analysis in a time window between 90–120 ms after the stimulus onset (referred to as the PA component). The timing of this component corresponds to the timing of the earliest positive peak of visual ERPs observed in previous studies [67, 68]. As visual areas are known to have a higher sensitivity to the contralateral stimuli [69], we also tested the responses of the posterior ROI to contralateral stimuli (i.e., responses of O1 and PO7 to the stimuli on the right visual hemifield and O2 and PO8 to the stimuli on the left visual hemifield) and measured the correlation of contralateral responses with behavior (see S5 and S7 Figs). Lastly, P3 responses (300–600 ms) were also inspected in the posterior ROI (O1, O2, PO7, PO8) in addition to the midline electrodes as visual inspection suggested a more posterior topography at this time interval than expected [70]. Statistical analysis We used 2 by 2 repeated measures ANOVAs (RM ANOVAs) to test the effect of Reward Value (high, low) and Modality (visual, auditory) on behavioral (d’ and RT) and electrophysiological responses during the associative reward learning and orientation discrimination tasks. Planned, paired t-tests were used for pairwise comparisons. Effect sizes in RM ANOVAs are reported as partial eta-squared (ηp2) and in pairwise comparisons as Cohen’s d; i.e. dz [71]. Before applying the RM ANOVAs, the assumption of normality was confirmed by inspecting the histograms and Quantile-Quantile (Q-Q) plots of the data. We only observed small deviations from normality in some cases and therefore decided to proceed with our preregistered analysis plan. To remove the effect of perceptual biases that participants may have for different colors or tone pitches prior to the learning of cues’ reward associations, we corrected the behavioral and ERP results of each participant during the orientation discrimination task by subtracting the data of each condition in pre-conditioning from the data in post-conditioning. We note that this method differs from our pre-registered plan where we intended to test the pre-conditioning data separately and rule out significant differences between reward conditions prior to learning of reward associations. While our pre-registered plan was suited to test the “statistical significance” of such biases, it did not entirely remove their potential contribution to the effects that occurred after learning of reward associations. We therefore decided to employ a stricter correction for such biases and test post-conditioning results after subtraction of pre-conditioning data. The data of the conditioning phase is reported without such correction as the corresponding task was never performed in the absence of reward feedbacks. Results Conditioning phase Behavioral results Participants’ performance in the localization task employed during the conditioning was near perfect as accuracies for both cues were > 95% (99% ± 1% for auditory and 100%±0% visual cues). Analysis of reaction times (RTs) revealed a significant main effect of modality (F(1,35) = 70.44, P < 0.001, ηp2 = 0.67). Participants localized visual cues (mean ± s.e.m.: 501 ± 12 ms) faster than auditory cues (mean ± s.e.m.: 584 ± 16 ms), in line with the superior performance of vision in localization tasks [72, 73]. We did not observe a main effect of reward value (F(1,35) < 1); however, an interaction was found between reward value and modality (F(1,35) = 7.68, p = 0.009, ηp2 = 0.18). Post-hoc pairwise comparisons did not reveal a significant effect in either of the modalities: for auditory cues, high reward value slowed down responses (mean ± s.e.m.: 589 ± 17 ms and 580 ± 16 ms for high and low value cues, respectively; t(35) = 1.52, p = 0.137, dz = 0.253), while in the visual modality, high-value cues sped up responses (mean ± s.e.m.: 496 ± 13 ms and 506 ± 12 ms for high- and low-value cues, respectively; t(35) = -2.02, p = 0.051, dz = 0.34). ERP responses: P1, N1, P3 We next examined whether the posterior ROI (PO7/PO8 and O1/O2) that we had planned to use for probing reward effects during the visual orientation discrimination task exhibits reliable reward modulations (ERP amplitude and or latency) during the reward associative learning (Fig 2 and Table 1, see also S1 Fig). 10.1371/journal.pone.0287900.g002 Fig 2 ERP responses of the posterior ROI to the visual and auditory reward cues during the conditioning phase. a) ERPs of visual reward cues with high (red traces) and low (blue traces) values measured in a posterior ROI (O1, O2, PO7 and PO8). The shaded grey areas correspond to 30 ms windows around the peak of P1 (70–170 ms) and N1 (180–250 ms) used to estimate the amplitude of these components for each condition (here the window is averaged across high and low value conditions) and the window used to estimate P3 responses (300–600 ms). The topographic distribution of P1, N1 and P3 components are shown for each reward value condition. b) Same as a for auditory reward cues. c-d) analysis of the amplitude and latency of the ERP components revealed significant value-driven modulations for the amplitude of P3 shown in c and the latency of the N1 component shown in d. Visual High Value: VH; visual Low Value: VL; Auditory High Value: AH; Auditory Low Value: AL. 10.1371/journal.pone.0287900.t001 Table 1 Amplitude and latencies of visual ERP components evoked by visual high value (VH), visual low value (VL), auditory high value (AH), and auditory low value (LA) cues during the conditioning phase, measured in the posterior ROI. Amplitude (μV) P1 N1 P3 VH 2.26 ± 0.25 0.62 ± 0.33 2.85 ± 0.42 VL 2.27 ± 0.26 0.84 ± 0.33 2.74 ± 0.38 AH 3.65 ± 0.29 -1.22 ± 0.36 2.88 ± 0.27 AL 3.66 ± 0.31 -1.49 ± 0.39 2.56 ± 0.29 Latency (ms) P1 N1 VH 154.67 ± 3.03 208.47 ± 2.86 VL 155.86 ± 3.68 215.58 ± 2.99 AH 134.94 ± 3.30 218.47 ± 3.23 AL 126.56 ± 2.97 214.69 ± 3.24 P1 amplitudes were impacted by modality (F(1,35) = 12.89, p < 0.001, ηp2 = 0.27), with higher amplitudes for auditory (mean ± s.e.m: 3.66 ± 0.28 μV) than visual cues (mean ± s.e.m: 2.27 ± 0.24 μV). Effects of reward values (F(1,35) < 1, p = 0.919) and modality by reward interaction (F(1,35) < 1, p = 0.971) did not reach significance. Analysis of P1 latency revealed a main effect of modality (F(1,35) = 53.1, p < 0.001, ηp2 = 0.60), reflecting an earlier P1 peak for auditory (mean ± s.e.m.: 130.7 ± 2 ms) compared to visual cues (mean ± s.e.m.: 155.3 ± 2.7 ms). Neither a main effect of reward value (F(1,35) = 1.19, p = 0.284) nor an interaction between reward value and modality (F(1,35) = 2.72, p = 0.108) was found. Analysis of N1 amplitudes in the conditioning phase revealed a main effect of modality (F(1,35) = 17.42, p < 0.001, ηp2 = 0.33), again with larger amplitudes for auditory (mean ± s.e.m: -1.36±0.35 μV) compared to visual cues (mean ± s.e.m: 0.73±0.32 μV). Neither the main effect of reward value nor the reward by modality interaction reached significance (all ps > 0.1). N1 latency was not modulated by modality (F(1,35) = 1.92, p = 0.175) or by reward value (F(1,35) < 1, p = 0.477). However, a significant interaction was found between reward value and modality (F(1,35) = 7.07, p = 0.012, ηp2 = 0.17), reflecting the different directions of value-driven modulation of N1 latencies in the two modalities (Fig 2B). Whereas visual high value cues significantly sped up the N1 peak (mean ± s.e.m.: 208.47 ± 2.86 ms and 215.58 ± 2.99 ms and for high and low value cues respectively, t(35) = -2.54, p = 0.016, dz = 0.424), auditory high-value cues slightly slowed down the N1 responses, an effect that did not reach statistical significance (mean ± s.e.m.: 218 ± 3.3 ms and 214.7 ± 3.2 ms for high and low value cues respectively, t(35) = 1.12, p = 0.27, Cohen’s d = 0.19). Analysis of the P3 amplitude in the posterior ROI (PO7, O1, O2, PO8), quantified between 300 and 600 ms, revealed a main effect of reward value (F(1,35) = 4.89, p = 0.034, ηp2 = 0.123, Fig 2C), reflecting larger amplitudes for high-reward cues (Mean ± s.e.m.: 2.87 ± 0.29 μV) than for low-reward cues (Mean ± s.e.m.: 2.65 ± 0.3 μV). The main effect of modality (F(1,35) < 1, p = 0.816) and an interaction effect between modality and value (F(1,35) < 1, p = 0.458) were non-significant. Since learning of reward associations may take time, and behavioral and ERP effects of reward may arise only after associative learning has been completed, we wondered whether our reported results differed between the first and the second halves of the conditioning. To test this possibility, we divided the trials for each condition to two halves and entered an additional factor, i.e., phase (first or second half of conditioning), in all of our ANOVAs. We found similar effect sizes in each case (the interaction between reward and modality for RT: F(1,35) = 7.70, p = 0.009, ηp2 = 0.18 and for N1 latency: F(1,35) = 7.26, p = 0.011, ηp2 = 0.172, and the main effect of reward on P3 amplitude: F(1,35) = 4.27, p = 0.046, ηp2 = 0.109), but we did not observe a significant interaction with the phase (all ps>0.1). Therefore, our reported results did not show a dependence on the phase of conditioning, probably because full learning of the reward associations was achieved very fast. In summary, examination of the posterior ROI revealed a significant interaction of reward value and reward modality on the latency of N1 responses, with a significant speeding up of ERP responses to high-value compared to low-value visual cues. In the P3 time window, a main effect of reward value was found across all high- compared to low-reward value conditions. Post-conditioning phase Behavioral results To assess the behavioral visual sensitivity, participants’ d-prime scores (d’ Post minus d’ Pre) were subjected to an ANOVA with reward value (high or low) and modality (intra- or cross-modal) as independent factors (Table 2 and Fig 3). This analysis revealed no main effect of modality (F(1,35) < 1, p = 0.99) or reward value (F(1,35) < 1, p = 0.35). Importantly, an interaction was found between modality and reward value (F(1,35) = 5.75, p = .022, ηp2 = 0.14, Fig 3E). Whereas cross-modal reward value significantly enhanced the visual sensitivity (mean ± s.e.m: 0.44 ± 0.19, for the difference between high and low value stimuli, corrected for pre-conditioning, t(35) = 2.31, p = 0.027, dz = 0.38, Fig 3E), intra-modal reward value produced an insignificant decrement in visual sensitivity (mean ± s.e.m.: -0.18 ± 0.19, for the difference between high and low value stimuli, corrected for pre-conditioning, t(35) = -0.97, p = 0.34, dz = 0.162). Analysis of the reaction times (RT) revealed an overall slowing down of responses for high- compared to low-value stimuli, but this effect did not reach statistical significance (F(1,35) = 2.36, p = 0.133). Other main and interaction effects were non-significant (all Fs < 1, all ps > 0.1). 10.1371/journal.pone.0287900.g003 Fig 3 Behavioral performance in the orientation discrimination task. a) Visual sensitivity (d-prime: d’) during the pre-conditioning phase for different conditions (Neutral: Neut; Intra-modal High Value: IH; Intra-Modal Low Value: IL; Cross-modal High Value: CH; and Cross-Modal Low Value: CL). b) Same as a, for reaction times (RT). c) Visual sensitivity during the post-conditioning phase, i.e. after the associative reward value of different cues were learned. d) Same as c for RTs. e) Effect size for d’ modulations, measured as the difference in d′ between high- and low-value conditions corrected for their difference during pre-conditioning, in intra- and cross-modal cue types (grey and white bars, respectively). Each dot represents the effect size for one individual subject. f) Same as e for reaction times (RT). * marks the significant effects (p < 0.05). Error bars are s.e.m. 10.1371/journal.pone.0287900.t002 Table 2 Summary of behavioral and ERP data during the pre- and post-conditioning phases (the gray and white cells, respectively) for intra-modal (IH and IL, high and low values), cross-modal (CH and CI, high and low value), and neutral (Neut) conditions. Significant pairwise comparisons are shown in bold fonts (p < 0.05). Pre-conditioning Post-conditioning IH IL CH CL Neut IH IL CH CL Neut RT (ms) 950 ±25.3 968 ±28.2 955 ±28 961 ±27.8 955 ±29.7 923 ±21.7 917 ±23.4 930 ±22.6 922 ±20.9 906 ±22.8 d’ 2.18 ±0.17 2.18 ±0.18 1.94 ±0.16 2.24 ±0.21 2.05 ±0.16 1.89 ±0.13 2.06 ±0.14 1.96 ±0.16 1.82 ±0.14 1.81 ±0.14 PA (μV) 0.62 ±0.29 0.16 ±0.26 3.52 ±0.40 3.89 ±0.41 0.09 ±0.24 0.09 ±0.15 0.39 ±0.17 3.36 ±0.34 3.30 ±0.36 0.40 ±0.17 P1 (μV) 2.28 ±0.36 2.35 ±0.35 5.56 ±0.43 6.03 ±0.39 1.98 ±0.34 1.84 ±0.26 2.07 ±0.30 4.81 ±0.34 5.00 ±0.36 1.93 ±0.26 N1 (μV) 0.93 ±0.41 0.99 ±0.39 -1.29 ±0.46 -0.67 ±0.48 0.42 ±0.42 0.38 ±0.35 0.74 ±0.35 -0.70 ±0.40 -0.94 ±0.41 0.91 ±0.36 P3 (μV) 2.65 ±0.41 2.70 ±0.43 3.44 ±0.47 4.23 ±0.49 1.98 ±0.46 2.58 ±0.46 2.48 ±0.45 3.91 ±0.47 3.76 ±0.49 2.52 ±0.45 P1 (ms) 151 ±3.63 152 ±3.78 141 ±3.08 145 ±2.78 147 ±4.15 147 ±4.01 147 ±4.33 140 ±3.62 136 ±3.50 151 ±3.26 N1 (ms) 214 ±3.85 213 ±3.71 214 ±2.82 213 ±2.81 211 ±3.59 209 ±3.29 215 ±3.34 213 ±2.66 210 ±2.33 212 ±3.65 In summary, during the post-conditioning phase, only cross-modal high-value cues significantly improved the visual sensitivity of orientation discrimination. Reaction times were not significantly affected by the reward value of either type. ERP responses in the posterior ROI: Pre-registered analyses Examination of P1 amplitudes with a two-way RM ANOVA revealed a main effect of modality (F(1,35) = 5.65, p = 0.023, ηp2 = 0.14), corresponding to larger P1 amplitudes in the cross-modal condition than in the intra-modal condition (see Table 2). The main effect of reward value (F(1,35) < 1, p = 0.831) and the interaction between reward value and modality did not reach significance (F(1,35) < 1, p = 0.501). Analysis of P1 latencies did not reveal any effect of modality (F(1,35) < 1, p = 0.935), value (F(1,35) < 1, p = 0.342), or their interaction (F(1,35) = 1.2, p = 0. 287). Analysis of the N1 amplitude revealed a trend for an effect of modality (F(1,35) = 3.6, p = 0.066, ηp2 = 0.093), with intra-modal reward cues evoking less negative N1 peak than cross-modal reward cues (see Table 2). The main effect of reward value did not reach significance (F(1,35) = 2.04, p = 0.162) but a trend was found for an interaction between reward value and modality (F(1,35) = 4.09, p = 0.051, ηp2 = 0.105), where high value cross-modal cues decreased N1 negativity compared to intra-modal condition (see Table 2, Figs 4 and 5 and S4 Fig). Planned, pairwise comparisons showed that the cross-modal high value condition significantly decreased the N1 negativity compared to the low value condition (t(35) = 2.98, p = 0.005, dz = 0.5, Fig 5C). The difference between intra-modal, high and low value conditions did not reach statistical significance (p = 0.45). Latency analysis of N1 component did not reveal any effects of modality (F(1,35) < 1, p = 0.935), value (F(1,35) < 1, p = 0.363) or their interaction (F(1,35) = 2.17, p = 0.150). 10.1371/journal.pone.0287900.g004 Fig 4 ERP responses of the posterior ROI (PO7, PO8, O1, O2) during the visual orientation discrimination task. a) ERPs elicited by the neutral condition during pre- and post-conditioning (illustrated on the left and right, respectively). The topographic distributions of PA (90–120 ms), P1 (most positive peak 70–170 ms), N1 (most negative peak 180–250 ms), and P300 (300–600 ms) components, measured in respective grey shaded areas of the ERP time courses are also shown. b) ERPs in the presence of task-irrelevant, intra-modal reward cues, with high (red traces) and low (blue traces) values. The corresponding topographic distribution of each component is shown for each reward value condition. To test the significance of value-driven modulations, the difference between high- and low-value conditions was corrected for their pre-conditioning difference. The topographic distributions of these corrected modulations are shown for each component (the lowermost topographic maps). See also Fig 5C and S4 Fig. c) Same as b for ERPs in the presence of task-irrelevant, cross-modal reward cues. 10.1371/journal.pone.0287900.g005 Fig 5 Mean amplitude of different ERP components of the posterior ROI (PO7, PO8, O1, O2) during the visual orientation discrimination task. a) mean amplitude of PA, P1, N1 and P3 responses evoked by neutral (N), intra-modal high value (IH), intra-modal low value (IL), cross-modal high value (CH) and cross-modal low value (CL) conditions, during pre-conditioning. b) Same as a, for ERPs during post-condoning. c) Corrected effect sizes, measured as the difference between high- and low-value conditions corrected for their difference during pre-conditioning, for intra- and cross-modal cue types (grey and white bars, respectively). * mark significant differences (p< 0.05) and ** mark significant differences (p< 0.01). Each dot represents the effect size for one individual subject. Error bars are s.e.m. Based on our pre-registered analysis plan, we next examined the earliest time points where the reward modulations of each cue type reached significance using a moving window analysis (see Methods). This analysis revealed an earlier onset of value-driven modulations for intra-modal compared to cross-modal conditions in the posterior ROI. High- and low-value ERP responses of the intra-modal conditions differed significantly within the P1 window, i.e. between 101 and 114 ms. The cross-modal value-modulations started later, at 167 ms, and remained significant until 223 ms, thus overlapping with the N1 time window (Fig 4). P3 responses at midline electrodes Analysis of the P3 component in midline electrodes (300–600 ms, corrected for pre-conditioning) revealed no main effect of modality, reward value, electrode, or their interaction (all ps > 0.1). Although inspection of the ERP responses of midline electrodes (see also S2 and S3 Figs) suggests that value-driven modulations occur in other time windows than our pre-registered intervals, we decided to adhere to our a priori plan and do not explore these modulations any further. ERP responses in the posterior ROI: Exploratory analyses Based on the timing of the earliest positive peak of visual ERPs (i.e. 90–120 ms) observed in classical studies of visual perception [67, 68], we next performed an exploratory analysis on the amplitude of evoked responses in this time window (referred to as the PA component throughout, see Figs 4 and 5 and S4 Fig). A two-way ANOVA on PA amplitudes revealed a significant value × modality interaction effect (F(1,35) = 5.12, p = 0.03, ηp2 = 0.128), but no main effect of reward value or modality (both Fs < 1 and ps > 0.1). Post-hoc pairwise comparisons (Fig 5C) revealed a significant suppression of PA amplitude for high- compared to low-value intra-modal condition (t(35) = -2.1, p = 0.04, dz = 0.35). The value-driven modulation in the cross-modal conditions was in the opposite direction but did not reach statistical significance (t(35) = 1.35, p = 0.185, dz = 0.22). We wondered whether the negative modulation of PA responses by intra-modal reward value reflects a suppression of high-reward stimuli or rather an enhancement of the low-reward conditions. To this end, we contrasted the PA amplitudes of intra-modal high- and low-value conditions, respectively, against the neutral condition that had not undergone reward associative learning (see also Figs 4 and 5). These comparisons revealed a significant suppression of PA responses evoked by intra-modal, high-value compared to neutral stimuli (t(35) = -2.07, p = 0.04, dz = 0.35). The intra-modal, low-value stimuli, however, were not significantly different from the neutral condition (t(35) = 0.25, p = 0.80, dz = 0.042). These results suggest that intra-modal high-value stimuli undergo active suppression compared to both intra-modal low value and neutral conditions occurring very early after the onset of stimuli. We next inspected the ERPs of the posterior ROI in a later time window (300–600 ms) corresponding to the P3 component (see Figs 4 and 5 and S4 Fig). This analysis revealed a main effect of reward value (F(1,35) = 5.63, p = 0.023, ηp2 = 0.138). The main effect of modality and the modality by reward interaction were not significant (both ps > 0.1). Post-hoc pairwise comparisons revealed a significant enhancement of P3 responses by cross-modal high- compared to low-value condition (t (35) = 2.72, P = 0.01, d = 0.45). In the intra-modal condition, P3 amplitudes did not differ between the two reward conditions (t(35) = 0.37, p = 0.72, dz = 0.061). Thus, examination of late ERPs in the posterior ROI overall indicated a value-driven response enhancement for the cross-modal condition (Fig 5C). Taken together, the ERP effects provide evidence for an interaction effect between intra- and cross-modal reward cues, with intra-modal cues leading to an early suppression and cross-modal reward cues producing a later response enhancement for high- compared to low-value stimuli. Importantly, we excluded the possibility that these effects were driven by changes in eye position in response to different reward conditions (see S1 Text). Similar results were obtained when contralateral ERPs were examined or when only correct trials were included in our analysis (see also S1 Text, S5 and S6 Figs). Correlation of behavioral and ERP amplitudes We next measured the correlation between value-driven modulations of ERP amplitudes for which a significant reward effect was found (i.e., PA, N1 and P3, see Fig 5) and behavioral indices (d’ and RT). This pre-registered analysis did not reveal any significant correlation (all ps> 0.1). However, an exploratory analysis revealed a significant positive correlation between value-driven modulations of the contralateral N1 and P3 components and behavioral d-primes in cross-modal stimuli (see S7 Fig). Discussion In the current study we tested whether the perception of a compound stimulus comprising a visual target and task-irrelevant intra-modal or cross-modal cues is affected by the reward associations of task-irrelevant components. To this end, we examined behavioral and electrophysiological responses to reward cues during a conditioning phase when reward associations were learned and during a post-conditioning phase when rewards were not delivered anymore and reward cues were irrelevant to the visual discrimination task that participants had to perform. In the conditioning phase, we found that intra- and cross-modal reward cues affect latency of the N1 responses over the posterior electrodes differently: while intra-modal reward cues significantly sped up N1 response, cross-modal reward cues only led to an insignificant deceleration of N1 responses. However, an exploratory analysis of P3-like responses of the posterior ROI revealed higher amplitudes for both intra- and cross-modal high value cues. In the post-conditioning phase, similarly to conditioning phase, P3-like responses of the posterior ROI were enhanced for high value cues of both types. Importantly, the effect of intra- and cross-modal rewards on earlier components of posterior ERPs differed. Cross-modal stimuli led to a later reward modulation at a time point that corresponded to the N1 component, whereas intra-modal reward modulations occurred earlier and an exploratory analysis revealed that high reward cues significantly suppressed posterior ERPs relative to low reward cues within 90–120 ms (PA window). Behavioral results were in line with electrophysiological findings; cross-modal reward cues significantly enhanced visual sensitivity whereas intra-modal reward cues led to a weak suppression. Interaction effects of reward value with modality within PA and N1 windows suggest that intra-modal and cross-modal reward values exert different effects on perception of a compound object under settings employed in our study (i.e., task-irrelevant reward cues during a no-reward phase). Similar reward effects of intra- and cross-modal conditions in the P3 window, observed in our exploratory analysis, indicate that beyond the differential effect of modality on the early components of posterior ERPs, the later reward effects are independent of the sensory modality. Reward-driven modulations during conditioning In this phase, a visual target predictive of higher rewards sped up reaction times and early cue-evoked neural responses, particularly in the N1 window. This result is similar to early modulations of visual ERPs (i.e. <250 ms) observed in previous studies [15, 17, 54, 70, 74, 75]. Auditory stimuli also evoked strong responses in the posterior ROI even in the absence of concomitant visual stimulation [76–79]. However, reward signals from auditory stimuli did not significantly affect the amplitude or latency of early ERP components when auditory stimuli were presented alone. Lack of cross-modal reward effects during the conditioning phase may indicate that reward information does not transfer automatically when there is no incoming visual information. The similarity of intra- and cross-modal reward effects on later P3 component, observed in an exploratory analysis, indicates that at the later stages of sensory processing reward information integrates into a coherent reward representation across various sources irrespective of the unique characteristics of those sources. On the other hand, during the early stages of sensory processing, privileged processing of rewarded stimuli only prioritizes cues that are most suited for the task at hand, i.e., vision in case of a task requiring localization. These results are overall explainable within the framework of reinforcement learning, where association of a stimulus with reward results in value-driven modulations at 2 time points [74]: an early modulation of neural responses for stimuli associated with higher value within the first 200–250 ms after the stimulus onset [54, 74]; and a later reward modulation primarily the P3 component related to anticipation of the reward delivery [17, 74, 80–82]. Reward-driven modulations during post-conditioning The results obtained in the post-conditioning phase showed a differential pattern of reward-related modulation for cross-modal and intra-modal cues. Specifically, we observed an improvement of behavioral measures of visual sensitivity for high compared to low-value cross-modal cues. These results are in line with the reported facilitatory effects of cross-modal value on visual processing observed in human psychophysical and neuroimaging studies [21, 61]. As a first attempt to characterize the electrophysiological correlates of this effect, our study demonstrated that the behavioral advantage conferred by high-reward auditory cues is accompanied by a modulation of posterior ERPs. As these modulations were corrected for differences potentially occurring due to physical stimulus features (i.e., tone pitches) already during the pre-conditioning phase, they most probably reflect effects driven by the associative learning of the reward values. However, we note that overall, the effect sizes in our study were small, likely due to the fact that reward cues were not the task-relevant features of the stimulus and did not predict the delivery of reward anymore, factors that can be further investigated in future studies. Cross-modal reward modulations in our study occurred later than attentional effects observed in some of the previous studies of cross-modal attention [83], where response modulations were found in P1 window. However, in these experiments auditory tones were presented prior to the visual target and could therefore modulate the early ERP responses. In fact, when auditory and visual components of an audiovisual object were presented together, attentional effects occurred at a later time window around 220 ms [84, 85]. Whereas reward-driven boost of attention may to some extent account for cross-modal reward effects in our study, the direction of response modulations suggests that additional mechanisms may also be involved (Fig 5C). Specifically, we found a reduction in N1 negativity after learning of the reward associations, which is different from an enhanced P1 positivity and N1 negativity that has been observed in studies of cross-modal attention [83–85]. One possible mechanism is a reward-driven enhancement of audiovisual integration, which is in line with recent findings demonstrating a role of reward in multisensory integration [44, 45]. This mechanism can also explain the direction of ERP modulations, as previous studies found that an audiovisual stimulus elicits response modulations of visual ERPs mainly in N185 window, with a reduction of the negativity of N185 component compared to the unimodal stimuli [64], which is similar to the pattern of modulations we observed for cross-modal rewards. The reduction in N1 negativity may indicate that audiovisual integration enhances the gain of the neural responses of visual cortex, hence reducing the energy (i.e. N1 amplitude) needed to process the same load of sensory information, an example of sub-additive cross-modal interactions reported before [48, 86, 87]. An enhanced integration between auditory reward cues and visual target in our task potentially reduces the distracting effect of task-irrelevant sounds on visual discrimination while promoting the spread of privileged processing from rewarded sounds to the visual target. Later modulations by cross-modal compared to intra-modal rewards suggests that a putative reward-driven enhancement of audiovisual integration may first occur in multimodal areas such as in Superior Temporal Sulcus (STS), being then fed back to visual cortex, a proposal in line with the findings of neuroimaging studies of cross-modal value and emotion effects on vision [21, 88]. In addition to the above mechanisms (reward-driven boost of attention and audiovisual integration), the later value-driven modulations in P3 window found in an exploratory analysis may indicate that the cross-modal reward effects also rely on post-sensory and decisional stages [89]. The effects of intra-modal reward cues on visual perception were contrary to our a-priori hypothesis that intra-modal and cross-modal reward stimuli should have similar facilitatory effects on perception of a compound object. This hypothesis was based on previous findings [11] that reported a spread of reward enhancement effects from one component of an object to its other parts (here from colored circles signaling reward to the Gabor target), akin to the spread of object-based attention [47, 90]. Contrary to our prediction, our exploratory analysis revealed that intra-modal reward stimuli interfered with sensory processing of the visual target, which was reflected in suppressed ERP responses at an early time window of 90–120 ms elicited by high rewards compared to both low reward and neutral conditions. The interference effect that we observed is in line with the findings of several studies where value-driven effects during visual search were investigated [30, 91–99]. These experiments have consistently reported that presentation of a high reward stimulus at the target location speeds up visual search and enhances target-evoked responses, whereas presentation of high reward stimuli at the distractor location captures attention away from the target and interferes with the search task. Given these previous findings, we superimposed reward cues on the target to boost the processing of all object elements at that location. However, since the visual discrimination task was performed on a different feature of the object (i.e., orientation) than the defining feature of the reward cue (color), high reward visual cues may have captured attention away from the target feature, hence interfering with the target processing. A similar interference effect has been observed in studies where a certain feature of an object was predictive of its reward but this feature was incongruent with the goal of the task, e.g. in many tasks employing the Stroop Effect [100–104]. In the light of these previous studies, we propose three possible mechanisms for the observed intra-modal, reward-related suppression. Firstly, it is possible that as a result of an enhanced response to the high-reward, task-irrelevant cues [105] some form of local inhibition is exerted on the adjacent stimuli, thereby decreasing the overall responses to target and rewarding cues that were at the same location. Such a center-surround inhibition around an attended feature has been observed in studies of feature-based and spatial attention [106–108]. Secondly, it is possible that the suppression is a reflection of the higher processing load of high-reward cues [109] and the capacity limitation of attentional processing. This mechanism could therefore result from a mixture of enhanced processing of task-irrelevant reward cues in some trials and decrement of processing of target+task-irrelevant cues in other trials due to the depletion of attentional resources. Thirdly, it is possible that the suppression is due to cognitive control mechanisms that actively inhibit the processing of intra-modal reward cues with a resultant spillover of inhibition to the target as also observed in previous studies of feature-based attention [110]. Overall, the above scenarios all indicate that the privileged processing of high reward intra-modal cues (as shown during the conditioning) did not effectively spread to the visual target. Therefore, it is possible that our paradigm failed to promote the integration of task-irrelevant intra-modal reward cues and the target into a coherent object. This issue can be remedied by increasing the duration of training on the task or using more object-like stimuli. Future studies will be needed to tease apart these possibilities. Taken together, our study demonstrates that the perception of an object can be influenced by the reward value of its constituent components both intra- as well as cross-modally. The differences that we observed between the intra-modal and cross-modal conditions might be due to the specific features of our experimental design (e.g., task-irrelevant cues that were previously associated with rewards), and some of our reported effects were observed while employing exploratory analyses and hence they await future replications and extensions. In this vein, our results also indicated that the perceived intensity of visual and auditory stimuli might have been different, as both P1 and N1 components were stronger for auditory compared to visual stimuli during the conditioning phase. Since we quantified the ERP measures separately for each sensory modality, calculated the reward effects against stimuli from the same modality, and corrected for pre-conditioning biases, it is unlikely that our reported results are due to the differences in perceived intensity of auditory and visual stimuli. We note however that using an adaptive method to equalize the perceived intensity of auditory and visual stimuli [111] would be an interesting addition to future studies. Despite these limitations, the differential modulations by the intra-modal and cross-modal reward cues may point to distinct neural pathways mediating the effects of reward from the same or different sensory modality on visual perception, a possibility that can be further examined by employing methods with a better spatial resolution. Embedding cross-modal rewards in visual tasks is a promising tool to assist vision, especially in the face of visual impairments, through boosting cross-modal advantages conferred by another intact sensory modality and comparing reward effects across intra-modal and cross-modal cues provides a first critical step towards this aim. Supporting information S1 Text (DOCX) Click here for additional data file. S1 Fig ERPs of midline electrodes during the conditioning phase. (DOCX) Click here for additional data file. S2 Fig ERPs of midline electrodes during the pre-conditioning phase. (DOCX) Click here for additional data file. S3 Fig ERPs of midline electrodes during the post-conditioning phase. (DOCX) Click here for additional data file. S4 Fig This figure should be compared to Fig 4 in the main text. (DOCX) Click here for additional data file. S5 Fig Contralateral responses of the posterior ROI during pre- and post-conditioning phases, see also Fig 4 in the main text. (DOCX) Click here for additional data file. S6 Fig ERP results of the posterior ROI when only correct trials were included, see also Fig 4 in the main text. (DOCX) Click here for additional data file. S7 Fig Correlation between electrophysiological and behavioral effects of reward value. (DOCX) Click here for additional data file. We thank Adem Saglam for his help with programming of the experiment, Franziska Ehbrecht for her help with the data collection, and Jessica Emily Antono for her valuable comments on the manuscript. 10.1371/journal.pone.0287900.r001 Decision Letter 0 Megna Nicola Academic Editor © 2023 Nicola Megna 2023 Nicola Megna https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version0 19 Dec 2022 PONE-D-22-27111Differential effects of intra-modal and cross-modal reward value on perception: ERP evidencePLOS ONE Dear Dr. Pooresmaeili, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. The main points to be reviewed are the clarification of the research design and some substantial modifications required in exposing significant and non-significant effects. In particular, we ask why a visual neutral reward cue was used, which was not done for acoustic rewards. This point is essential to remove any doubts regarding the possible influence of this factor on the results. Greater clarity is required in distinguishing results that are significant from those that indicate a trend, from those that are statistically insignificant. Furthermore, some concerns are indicated about the normalization process to remove perceptual biases due to color or tone perception. We also remind you that PLOS ONE has a specific policy regarding data availability (see https://journals.plos.org/plosone/s/data-availability). We understand that the authors have given the possibility to consult the data on request during the peer-review process, and that all data will be available after acceptance (in the form you report: "the URLs/accession number/DOIs will be available only after acceptance of the manuscript for publication so that we can ensure their inclusion before publication"). We ask for confirmation. Please submit your revised manuscript by Feb 02 2023 11:59PM. This is the standard date revision due. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript:A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Nicola Megna, M.D. Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and  https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.  In your revised cover letter, please address the following prompts: a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent. b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. We will update your Data Availability statement on your behalf to reflect the information you provide 3. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide 4. "Please update your submission to use the PLOS LaTeX template. The template and more information on our requirements for LaTeX submissions can be found at " ext-link-type="uri" xlink:type="simple">http://journals.plos.org/plosone/s/latex." 5. Thank you for stating the following in the Acknowledgments Section of your manuscript:  "We thank Adem Saglam for his help with programming of the experiment, Franziska Ehbrecht for her help with the data collection, and Jessica Emily Antono for her valuable comments on the manuscript. This work was supported by an ERC Starting Grant (no: 716846) to AP. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript" We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.  Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:  "This work was supported by an ERC Starting Grant (no: 716846) to AP. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." Please include your amended statements within your cover letter; we will change the online submission form on your behalf [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Partly Reviewer #2: Partly ********** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: No Reviewer #2: No ********** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: No Reviewer #2: No ********** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ********** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: In this study, the authors investigated the effect of rewarding intra-modal or cross-modal cues on visual orientation discrimination using a combination of psychophysics and EEG. From a behavioral point of view, they found that highly rewarded sounds improved visual discrimination sensitivity whereas highly rewarded visual cues tended to interfere with discrimination performance. EEG data strengthened this dissociation: while highly rewarded visual cues elicited an early suppression of ERPs of posterior electrodes, highly rewarded sound cues produced an enhancement of amplitude of later (from N1 to P3) responses relative to sound cues associated with lower reward. The authors concluded that, while the reward magnitude can modulate sensory processing, it does so differently based on the involved sensory modalities. This is a very interesting study addressing a novel research question. The paper is also mostly well written. ERP analyses and the corresponding drawn conclusions look reasonable to me, but I do not have expertise with EEG, therefore I hope other reviewers could evaluate more deeply this part of the study and support that it is appropriate. However, I do have some concerns related to other methodological and data analysis aspects which I feel should be addressed before recommending this work for publication. MAJOR CONCERNS For clarity reasons, I report my major concerns as three different points. However, I believe they are all interrelated. - If I correctly understood the study design, the authors have included a neutral visual condition (grey cue) whereas a corresponding neutral auditory condition is missing. I was wondering why the authors have opted for this unbalanced design which may, in my opinion, affect the results in different ways. For instance, the probability of an intramodal cue is higher than the probability of a cross-modal cue which may influence participants’ expectation and, doing so, also the performance in the task. - The authors analyzed data using RM ANOVAs with Reward value (high, low) as one of the two factors. However, the study design included a neutral condition which has not been compared to the reward conditions. I was wondering why the authors did not include a Condition factor in their analyses with three levels (high reward, low reward, neutral). - The authors normalized data to remove the effect of perceptual biases that subjects may have for different colors or tone frequencies. This is, in my opinion, a correct procedure since there is evidence that, for instance, RT may be influenced by stimulus color even for isoluminant stimuli. However, the normalization procedure does not look appropriate to me. If I correctly understood, the authors subtracted the data of each condition in pre-conditioning from its counterpart in post-conditioning. This procedure allows to select the effect of reward but it does not eliminate the bias for specific chromaticies or tone frequencies. In my opinion, a possible correct method, given the unbalanced design, would be identifying each subjects bias in pre-conditioning (e.g., the difference between orange and blue in RTs, or the difference between high and low tone frequency) and correct the post-conditioning data by adding or subtracting the difference observed in pre-conditioning to limit if not eliminate the color (or pitch) bias. For instance, if a subject in pre-conditioning is 6 ms faster in the orange compared to the blue cue condition, 6 ms could be added in the orange condition post-conditioning data. In this way, the remaining difference between orange and blue in post-conditioning may be due to different reward value (plus noise). After this correction, the authors could implement the ANOVAs as defined. MINOR CONCERNS - The experimental procedures section requires more details. For instance, the authors did not explain what kind of calibration procedure they used and for how long it lasted. Similarly, they did not specify the number of sessions they used in the QUEST method as well as the number of trials. As for the conditioning phase, it is not clear what happened after a wrong response. What kind of feedback did the participants receive in this situation? Finally, did the authors checked for data normality? - Rows 359-360: please report p-values of t-tests. - Rows 392-394: the authors stated that auditory high-value cues tended to slow down N1 responses. I think the analysis does not support this statement since the p-value (0.27) is more compatible with a lack of difference. We usually indeed refer to a trend for p-values comprised between 0.05 and 0.10. - In general (e.g., Table 2), the authors used acronyms such as IH and CL also for the pre-conditioning conditions. This may be confusing for the reader since the cues were not conditioned yet. Perhaps, I would clarify this aspect in the manuscript. - Figure 3E: what does the double asterisk mean? - FIGURES 2,3,5: the authors are using a color code for bars which is not defined in the captions. - The manuscript requires some editing. References are not consistently formatted (e.g., row 73) and there are many language mistakes (e.g., ‘auditory tones’, row 177: ‘different’ instead of ‘difference’) and typos (e.g., caption Figure 5: ‘pre-condoning’; row 645: ‘enchantment’; row 664: ‘at a the’) Reviewer #2: In this very interesting study, the authors aimed to investigate the effect of modality (acustic or visual) and value of a reward in an orientation discrimination task, using both psychophysical and electrophysiological measures. They found that a cross-modal highly rewarded cue is efficient in improving visual sensitivity, while intra-modal reward cues tend to interfere with a visual task. There are some concerns about the experimental design and the statistical analysis. MAJOR CONCERNS: - The authors comment continuously not significant data, and I think this can be correct when this is underlined. When you read the article you have the impression of solid results, but at the end of many paragraph you discover that we are talking about trends. Even, on line 394, we read of a trend only to discover that the p value is 0.27. This is clearly not a trend. This is particular evident also in lines 420-437. I ask the authors, to make everything clearer, to break down the significant effects identified and talk about them first, and then proceed to identify the trends, and then reformulate everything. - The value of reward is not well balanced in the visual and auditory modality: there are three values for the visual cues (high-neutral and low value) and only two for the auditory cues (low and high). This is an important issue to me, because the salience of visual cues may be decreased for this very reason. The problem is that the pejorative effect of visual cues could be due to this aspect of the experimental design. Authors should therefore at least explain why they used a neutral condition, why they also didn't present the data in the neutral condition, or better, describe also this condition (neutral visual cue). Naturally, they should convince readers that the pejorative effect of visual cues is not due to this factor. MINOR CONCERNS: - There are many confusing aspects: acronyms, such like IH, CH, etc could be confusing. The experimental design should be described with more clarity. Figure 1c doesn't help (I suggest to modify or eliminate it). The pitch function is not immediately clear, because at rows 211-213 the reader has to infer it. line 225: maybe it is "cues", not "stimuli". There are also some typos to be addressed. ********** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: Yes: Nicola Megna ********** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. 10.1371/journal.pone.0287900.r002 Author response to Decision Letter 0 Submission Version1 17 Feb 2023 Summary of the decision letter and our response* [*In the PDF version of our rebuttal letter, the editor’s and reviewers’ comments are in black, our responses are in blue, and the revised sections in the manuscript are marked in green. Please note that the line numbers we refer to in our responses reflect the line numbers in the manuscript with track changes.] Editor’s email: PONE-D-22-27111 Differential effects of intra-modal and cross-modal reward value on perception: ERP evidence PLOS ONE Dear Dr. Pooresmaeili, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. The main points to be reviewed are the clarification of the research design and some substantial modifications required in exposing significant and non-significant effects. In particular, we ask why a visual neutral reward cue was used, which was not done for acoustic rewards. This point is essential to remove any doubts regarding the possible influence of this factor on the results. Greater clarity is required in distinguishing results that are significant from those that indicate a trend, from those that are statistically insignificant. Furthermore, some concerns are indicated about the normalization process to remove perceptual biases due to color or tone perception. We also remind you that PLOS ONE has a specific policy regarding data availability (see https://journals.plos.org/plosone/s/data-availability). We understand that the authors have given the possibility to consult the data on request during the peer-review process, and that all data will be available after acceptance (in the form you report: "the URLs/accession number/DOIs will be available only after acceptance of the manuscript for publication so that we can ensure their inclusion before publication"). We ask for confirmation. We thank the editor and the reviewers for their positive evaluation of our work and for their insightful suggestions. In our revisions, we have carefully implemented all reviewers’ suggestions, which have made the manuscript much stronger than the original version. We have also uploaded the data and the analysis scripts to the Open Science Framework (OSF), as outlined in the cover letter and the manuscript. We hope that we have satisfactorily addressed all reviewers’ comments and our manuscript can be now accepted for publication in Plos One. Point-by-point response Comments of Reviewer 1 Reviewer #1: In this study, the authors investigated the effect of rewarding intra-modal or cross-modal cues on visual orientation discrimination using a combination of psychophysics and EEG. From a behavioral point of view, they found that highly rewarded sounds improved visual discrimination sensitivity whereas highly rewarded visual cues tended to interfere with discrimination performance. EEG data strengthened this dissociation: while highly rewarded visual cues elicited an early suppression of ERPs of posterior electrodes, highly rewarded sound cues produced an enhancement of amplitude of later (from N1 to P3) responses relative to sound cues associated with lower reward. The authors concluded that, while the reward magnitude can modulate sensory processing, it does so differently based on the involved sensory modalities. This is a very interesting study addressing a novel research question. The paper is also mostly well written. ERP analyses and the corresponding drawn conclusions look reasonable to me, but I do not have expertise with EEG, therefore I hope other reviewers could evaluate more deeply this part of the study and support that it is appropriate. However, I do have some concerns related to other methodological and data analysis aspects which I feel should be addressed before recommending this work for publication. We thank the reviewer for the overall positive evaluation of our work and the insightful suggestions which have greatly helped us to improve our manuscript. MAJOR CONCERNS For clarity reasons, I report my major concerns as three different points. However, I believe they are all interrelated. - If I correctly understood the study design, the authors have included a neutral visual condition (grey cue) whereas a corresponding neutral auditory condition is missing. I was wondering why the authors have opted for this unbalanced design which may, in my opinion, affect the results in different ways. For instance, the probability of an intramodal cue is higher than the probability of a cross-modal cue which may influence participants’ expectation and, doing so, also the performance in the task. We thank the reviewer for raising this point. As outlined below in detail, the neutral condition was never presented during the conditioning and participants did not associate the neutral cue with any reward. This important point is now clarified in lines: 221-224, 238-240, 273-278 of the manuscript. In fact, we think that the schema shown in the original version of Figure 1d might have been confusing with this respect since the neutral condition was shown next to the visual high and low value conditions, as if it also represented a certain stimulus-reward association. We have now separated the neutral condition from the rest of the conditions in Figure 1d and added an explanation about the reward associations to the figure caption that clarifies this issue (lines 307-310). We note, however, that including a neutral auditory condition would have produced a 3-by-2 balanced design - for three levels of reward (high, low and neutral) and two modalities (visual and auditory). We decided not to use this design based on the following reasons: Firstly, in our pilot experiments, participants had difficulties in identifying different auditory tones and in discriminating them based on their pitch. Therefore, having three rather than two different auditory tones would have made the learning of the stimulus-reward associations more difficult for the auditory stimuli. This is different from visual conditions where the discrimination of different stimuli based on their color and therefore the learning of stimulus-reward associations was relatively easy for the participants. Secondly, the main aim of the study was to characterize the effect of reward magnitude, which we could undertake by comparing the high versus the low reward condition of each modality. Towards this aim, testing the neutral condition was unnecessary. Nevertheless, the rationale for including a neutral condition was to have a setting, in which the responses to the visual target can be characterized, as we did for the results shown in Figure 3-5, without any influence of the reward associative learning. Importantly, the neutral condition consisted of a Gabor patch plus an overlaying circle as in the other conditions (see Figure 1), thus allowing to estimate the baseline visual discrimination performance and the evoked responses by the visual target in each participant. Thirdly, the perceptual discrimination threshold for all participants was determined by using stimuli that were identical to the neutral condition, i.e. a Gabor patch plus an overlaying grey circle. This again shows that our neutral condition basically represented the responses to the visual target whereas intra-modal (visual) and cross-modal (auditory) conditions represented how this response was changed by another dimension: i.e., the reward associated with each color or sound pitch. In general, we agree with the reviewer that visual conditions were more probable than the auditory conditions and hence the responses to them might have differed from the less frequent auditory conditions. We note, however, that each visual or auditory reward condition was compared against its counterpart with a different reward magnitude. For instance, while the responses to all the intra-modal (visual) stimuli might have been stronger or weaker than the cross-modal (auditory) stimuli because of the expectation effect that the reviewer mentioned, the difference between the high and low reward conditions of each modality is not likely to be affected by this factor. - The authors analyzed data using RM ANOVAs with Reward value (high, low) as one of the two factors. However, the study design included a neutral condition which has not been compared to the reward conditions. I was wondering why the authors did not include a Condition factor in their analyses with three levels (high reward, low reward, neutral). Thanks for this comment. As mentioned in response to the first point, the neutral condition was never associated with any reward value. Therefore, its inclusion in our analyses would deter from our main aim which was to characterize the interaction between the reward value and the sensory modality. Due to the considrations mentioned in response to point 1, our design did not include a neutral condition for the cross-modal stimuli. Hence, the RM ANOVA that the reviewer suggested could only be applied to the data of visual (intra-modal) conditions. However, the main aim of RM ANOVAs employed in this study was to test the effect of reward value (high, low) across different modalities (intra-modal and cross-modal). The 2 by 2 RM ANOVAs that we used is therefore the most straightforward and parsimonious statistical test to examine whether the effect of reward value differed between the intra-modal (visual) and cross-modal (auditory) conditions. - The authors normalized data to remove the effect of perceptual biases that subjects may have for different colors or tone frequencies. This is, in my opinion, a correct procedure since there is evidence that, for instance, RT may be influenced by stimulus color even for isoluminant stimuli. However, the normalization procedure does not look appropriate to me. If I correctly understood, the authors subtracted the data of each condition in pre-conditioning from its counterpart in post-conditioning. This procedure allows to select the effect of reward but it does not eliminate the bias for specific chromaticies or tone frequencies. In my opinion, a possible correct method, given the unbalanced design, would be identifying each subjects bias in pre-conditioning (e.g., the difference between orange and blue in RTs, or the difference between high and low tone frequency) and correct the post-conditioning data by adding or subtracting the difference observed in pre-conditioning to limit if not eliminate the color (or pitch) bias. For instance, if a subject in pre-conditioning is 6 ms faster in the orange compared to the blue cue condition, 6 ms could be added in the orange condition post-conditioning data. In this way, the remaining difference between orange and blue in post-conditioning may be due to different reward value (plus noise). After this correction, the authors could implement the ANOVAs as defined. Thanks for this comment. Please note that the association of colors and tone pitches with rewards was counter-balanced across participants, as mentioined in lines 264-267 of the manuscript and the legend to Figure 1. Specifically, we eliminated the effects of the physical characteristics of reward cues by counterbalancing colors (orange-O, blue-B) and sounds (350 Hz-L, 1050 Hz-H) across participants. Accordingly, there were 4 groups: (1) high-value O/L, low-value B/H, (2) high-value B/H, low-value O/L, 3) high-value B/L, low-value O/H, 4) high-value O/H, low-value B/L. In addition to the counterbalancing across participants, we additionally removed the potential effect of different colors and sound pitches by subtracting the data of pre-conditioing from post-conditioning. The procedure that we use is in fact statisticallay and numerically identical to the reviewer’s suggestion as explained in the example below: suppose that for a certain participant, RTs in response to the orange and blue colors were 300 ms and 400 ms, repectively, during the pre-conditioing. Therefore, here the participant is 100 ms faster in orange condition before the reward associations are learned. Suppose also that the RTs changed to 250 ms and 380 ms in the post-conditioing: here the participant is 130 ms faster in orange condition. Subtracting the data of pre- from post-conditioing adjusts the difference of orange versus blue from -130 ms in post-conditioing to -30 ms ((250-300) – (380-400)). So the 130ms-faster reaction time in orange is rectified with the adjustment to 30ms-faster RT, which is exactly the way that the reviewer suggested. MINOR CONCERNS - The experimental procedures section requires more details. For instance, the authors did not explain what kind of calibration procedure they used and for how long it lasted. Similarly, they did not specify the number of sessions they used in the QUEST method as well as the number of trials. As for the conditioning phase, it is not clear what happened after a wrong response. What kind of feedback did the participants receive in this situation? Finally, did the authors checked for data normality? We thank the reviewer for pointing these out. We have now added these details to the Methods section. Specificially: Re. Calibration (lines 196-199): We have added that “An experimental session started with a calibration procedure, where for each participant the luminance of two consecutively presented colors was adjusted until the perceived flicker between them was minimized and they became perceptually isoluminant (total duration of calibration 5min). The calibration was followed by a short training session for the orientation discrimination task (Number of trials = 36)”. Re. QUEST: We have now clarified that QUEST is an adaptive method that tries to find the best estimation of each participant’s psychophysical threshold by estimating the probability of each response given the stimulus intensity. Hence, the number of trials differed between the participants and depended on the variance of their responses. We have now included a short description of the QUEST method to lines 201-205. Re. Conditioning: Participants’ performance in this task was near perfect as mentioned in lines 392-394. In the rare event of errors, the feedback display informed the participants that the reward on that certain trial was zero (no feedback was provided regarding the accuracy of the decisions in any part of the experiment). We have now added this information to lines 242. Re. Assumption of normality for parametric tests. We have now explained our procedure in lines 373-376: “Before applying the RM ANOVAs, the assumption of normality was confirmed by inspecting the histograms and Quantile-Quantile (Q-Q) plots of the data. We only observed small deviations from normality in some cases and therefore decided to proceed with our preregistered analysis plan”. - Rows 359-360: please report p-values of t-tests. Thanks for pointing this out. In fact, the p-values were mentioned in the subsequent lines but for better clarity, we now provide them right after the description of the effects in each individual modality (now lines 399-404). - Rows 392-394: the authors stated that auditory high-value cues tended to slow down N1 responses. I think the analysis does not support this statement since the p-value (0.27) is more compatible with a lack of difference. We usually indeed refer to a trend for p-values comprised between 0.05 and 0.10. Many thanks for pointing this out. Our choice of the word “tended” was a poor choice and might have implied that we consider this effect a “statistical trend”, which was not intended. As mentioned in the preceding lines (now 430-432), we observed an interaction between Reward and Modality factors in their effect on the latency of the N1 responses: visual cues significantly sped up the N1 responses (p = 0.016) and auditory cues slowed down the responses (p = 0.27), but the latter effect did not reach statistical significance. We have now rephrased this part (lines 434-437). - In general (e.g., Table 2), the authors used acronyms such as IH and CL also for the pre-conditioning conditions. This may be confusing for the reader since the cues were not conditioned yet. Perhaps, I would clarify this aspect in the manuscript. Thanks for this comment. While we understand that these acronyms may be confusing, since in pre-conditioing they refer to the stimulus conditions that were not yet associated with the rewards, we decided to keep them. This is to underscore that our effects in post-conditioning were all corrected relative to their counterparts in pre-conditioing. To clarify the choice of acronyms for the readers, we have now added the following explanation to the Methods section (lines 265-273): “The association of each cue identity with the reward value was counterbalanced across participants. This means that each reward condition (high or low), comprised an equal number of instances where either the orange or the blue color or the 350 Hz or the 1050 Hz tone was associated with that reward magnitude. Therefore, when describing the effects during the orientation discrimination task, we will refer to all stimulus conditions with respect to the reward assignment that they acquired after the conditioning, although during the pre-conditioning these associations were not learned yet. Accordingly, these conditions in both pre- and post-conditioning will be referred to as: Intra-modal High Reward: IH, Intra-modal Low Reward: IL, Cross-modal High Reward: CH and Cross-modal Low Reward: CL, plus a neutral condition (referred to as Neut), which was never associated with any reward value (as shown in Figure 1).” - Figure 3E: what does the double asterisk mean? The double asterisk refers to the interaction effect between reward and modality and the post-hoc followup test for this effect, which showed a significant effect of reward only in the cross-modal condition. We have now clarified the meaning of these asterisks in Figure 3E by referring to them in the main text where the statistical tests are described (Lines 468 and 471). - FIGURES 2,3,5: the authors are using a color code for bars which is not defined in the captions. The color codes in Figure 2 are defined (red for high and blue for low reward) in the legend. In Figure 3 and Figure 5, color codes correspond to condition labels shown on the x-axis, which are described in the legend (lines 486-487 and lines 597-599 for Figure 3 and 5, respectively). - The manuscript requires some editing. References are not consistently formatted (e.g., row 73) and there are many language mistakes (e.g., ‘auditory tones’, row 177: ‘different’ instead of ‘difference’) and typos (e.g., caption Figure 5: ‘pre-condoning’; row 645: ‘enchantment’; row 664: ‘at a the’) Thank you for pointing these mistakes out. We have now corrected all formatting issues and language mistakes. Comments of Reviewer 2 Reviewer #2: In this very interesting study, the authors aimed to investigate the effect of modality (acustic or visual) and value of a reward in an orientation discrimination task, using both psychophysical and electrophysiological measures. They found that a cross-modal highly rewarded cue is efficient in improving visual sensitivity, while intra-modal reward cues tend to interfere with a visual task. Thank you for the positive evaluation of our work and for the extremely helpful suggestions, which we have carefully implemented as outlined below. There are some concerns about the experimental design and the statistical analysis. MAJOR CONCERNS: - The authors comment continuously not significant data, and I think this can be correct when this is underlined. When you read the article you have the impression of solid results, but at the end of many paragraph you discover that we are talking about trends. Even, on line 394, we read of a trend only to discover that the p value is 0.27. This is clearly not a trend. This is particular evident also in lines 420-437. I ask the authors, to make everything clearer, to break down the significant effects identified and talk about them first, and then proceed to identify the trends, and then reformulate everything. Many thanks for this helpful comment. We agree that we should have separated the signficiant and insignificant effects more clearly. We have now amended this problem by clearly stating which effects were statistically significant, correcting reporting errors due to a poor choice of words (p = 0.27, also see the response to reviewer 1 about this), and clarification of the statistical trends which did not reach significance. Accordingly, we have applied changes throughout the manuscript (including corrections to the abstract and the discussion, see for instance lines 428-429 and 462 of the Results, lines 642-643, 668-669 and 670-673 of the Discussion). - The value of reward is not well balanced in the visual and auditory modality: there are three values for the visual cues (high-neutral and low value) and only two for the auditory cues (low and high). This is an important issue to me, because the salience of visual cues may be decreased for this very reason. The problem is that the pejorative effect of visual cues could be due to this aspect of the experimental design. Authors should therefore at least explain why they used a neutral condition, why they also didn't present the data in the neutral condition, or better, describe also this condition (neutral visual cue). Naturally, they should convince readers that the pejorative effect of visual cues is not due to this factor. Thanks for this comment. We think that the schema presented in Figure 1d might have been misleading, giving the impression that the neutral condition is equivalent to a reward value equal to 0. In fact, the neutral condition was never presented during the conditioning phase and the participants did not associate the neutral cue with any reward. This important point is now clarified in lines: 226-227, 231-233 and 273-278. We have also corrected Figure 1d and the legend to this panel to clarify this issue. As explained in lines 273-278 of the revised manuscript (and the lengend to Fig.1E), the rationale to include the neutral condition was to have a setting in which the responses to the visual target can be characterized. Since the neutral condition consisted of a Gabor patch plus an overlaying circle, similar to all other conditions (see Figure 1), inspection of the results in neutral condition allowed us to have an estimation of the baseline visual discrimination performance and the evoked responses by the visual target in each participant, as shown in Figure 3-5, (which was otherwise impossible as the visual target and the reward cues were at the same spatial location). As also mentioned in response to Reviewer 1’s first comment, we agree with the reviewer that visual conditions were more probable than the auditory conditions and hence the responses to them might have differed from the less frequent auditory conditions. We note, however, that each visual or auditory reward condition was compared against its counterpart with a different reward magnitude. For instance, while the responses to all intra-modal (visual) stimuli might have been stronger or weaker than cross-modal (auditory) stimuli because of the expectation effect that the reviewer mentioned, the difference between high and low reward conditions of each modality is not likely to be affected by this factor. Due to these reasons we do not think that our reported intra-modal value effects are related to the neutral condition. MINOR CONCERNS: - There are many confusing aspects: acronyms, such like IH, CH, etc could be confusing. The experimental design should be described with more clarity. Figure 1c doesn't help (I suggest to modify or eliminate it). The pitch function is not immediately clear, because at rows 211-213 the reader has to infer it. line 225: maybe it is "cues", not "stimuli". There are also some typos to be addressed. Thank you for these very helpful suggestions that are now implemented in the revised manuscript. We have tried to increase the clarity of the manuscript and improve our description of the experimental design at multiple locations including but not limited to those suggested by the reviwer. We have also corrected some unintended errors (the topoplots in Figure 4A were swapped between the pre- and post-conditioing, which is now corrected). In the light of Reviewer 1’s comments, we decided to keep Figure 1d and add clearer explanations about our procedures to the legend. We have also corrected all the typos and linguistic mistakes, as far as we could see. Attachment Submitted filename: RebuttalLetter2.docx Click here for additional data file. 10.1371/journal.pone.0287900.r003 Decision Letter 1 Megna Nicola Academic Editor © 2023 Nicola Megna 2023 Nicola Megna https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version1 24 May 2023 PONE-D-22-27111R1Differential effects of intra-modal and cross-modal reward value on perception: ERP evidencePLOS ONE Dear Dr. Pooresmaeili, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Jul 08 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript:A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-emailutm_source=authorlettersutm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Nicola Megna, M.D. Academic Editor PLOS ONE Journal Requirements: Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #2: All comments have been addressed Reviewer #3: (No Response) ********** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #2: Yes Reviewer #3: Yes ********** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #2: Yes Reviewer #3: Yes ********** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #2: Yes Reviewer #3: Yes ********** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #2: Yes Reviewer #3: Yes ********** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #2: The authors have fully answered the doubts raised in my previous review. It is very clear now what they meant by "neutral condition" and that the related cue was not associated with any reward. The meanings of the various acronyms have also been clarified in an appropriate manner and also the statistical significance of the various results is now much more readable and understandable. Reviewer #3: This is a very interesting study investigating the effect of intra-modal and cross-modal value on visual perception and its neural correlates combining psychophysics and EEG. I have been involved in the review process after a first round of review, after the authors already addressed several comments. The study investigates an interesting question, using a solid experimental paradigm. The results are interesting and somehow unexpected, with potentially important implications for our understanding of how the sensory brain encodes reward value. The analysis of the data is sound, and the manuscript is overall well written. I only have some minor comments: 1-Since the main EEG results are based on the difference between pre and post-conditioning traces (which I think is a very intellingent way of analysing the data), I think it might be helpful to show two separate panels (perhaps in figure 5), the average trace of the difference between post and pre conditioning for HI LI and HC and LC conditions. I think that this would help the reader to appreciate better the results, to complete the information provided by the small maps at the bottom of Figure 4b and c. 2- I find the results that, during the conditioning phase, auditory stimuli elicit a stronger P1 component (larger amplitude and shorter latency) than visual stimuli very intriguing. Could the authors discuss this result? How do they interpret it? 3- During the conditioning phase, is there a difference between the first and the second half of the trials? If participants learn the association between a particular stimulus and either high or low reward, a difference between the two migh emerge later on during the experimental session. 4-Figure 3e-f and Figure 5c: individual subjects data could be overlayed on top of the bar plots so to better appreciate the interindividual variability of the effect. line 124: Figure 4d actually refers to Figure 2b, please correct the typo. ********** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #2: Yes: Nicola Megna Reviewer #3: No ********** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. 10.1371/journal.pone.0287900.r004 Author response to Decision Letter 1 Submission Version2 10 Jun 2023 Point-by-point response Comments of Reviewer 2 Reviewer #2: The authors have fully answered the doubts raised in my previous review. It is very clear now what they meant by "neutral condition" and that the related cue was not associated with any reward. The meanings of the various acronyms have also been clarified in an appropriate manner and also the statistical significance of the various results is now much more readable and understandable. We thank the reviewer for the very insightful suggestions during the first round of revisions which have greatly helped us to improve our manuscript and make it much stronger than the initial submission. Reviewer #3: This is a very interesting study investigating the effect of intra-modal and cross-modal value on visual perception and its neural correlates combining psychophysics and EEG. I have been involved in the review process after a first round of review, after the authors already addressed several comments. The study investigates an interesting question, using a solid experimental paradigm. The results are interesting and somehow unexpected, with potentially important implications for our understanding of how the sensory brain encodes reward value. The analysis of the data is sound, and the manuscript is overall well written. Thank you for the positive evaluation of our work and for the extremely helpful suggestions, which we have carefully implemented as outlined below. I only have some minor comments: 1-Since the main EEG results are based on the difference between pre and post-conditioning traces (which I think is a very intellingent way of analysing the data), I think it might be helpful to show two separate panels (perhaps in figure 5), the average trace of the difference between post and pre conditioning for HI LI and HC and LC conditions. I think that this would help the reader to appreciate better the results, to complete the information provided by the small maps at the bottom of Figure 4b and c. Thank you for this great suggestion. Since both Figure 4 and 5 already contain many dense panels, we decided to add the difference waves (for HI-LI and HC-LC) and their corresponding topographic distributions to the Supplementary Information (Supplementary Figure 4) and refer to it in the legend of Figure 4 and in the main text (lines 545 and 562). 2- I find the results that, during the conditioning phase, auditory stimuli elicit a stronger P1 component (larger amplitude and shorter latency) than visual stimuli very intriguing. Could the authors discuss this result? How do they interpret it? Many thanks for pointing this out. Auditory tones have overall shorter processing latencies compared to visual stimuli [1]. However, as the reviwer noted, it is indeed intriguing that in visual cortex, an area that is dedicated to the processing of visual stimuli, the amplitude of P1 responses is larger for auditory compared to visual stimuli. We think that this finding is due to the relative strength of the visual and auditory stimuli that we utilized: visual stimuli in our experiments were small, transparent colored circles (0.44° in diameter), whereas the auditory stimuli were played at a suprathreshold intensity (70 dB) so that participants could detect the tones very easily. In fact, a different approach would be to equalize the perceived intensity of the auditory and visual stimuli by measuring participants’ detection threshold in a 2AFC paradigm for both stimulus types, similar to previous psychophsical studies [2]. However, in our experience (unplublished data), participants can easily detect auditory stimuli even at very low intensities and therefore using an equalization method for visual and auditory stimuli may yield extremely low intensities for visual stimuli which would make the discrimination between them difficult. Since we measure response latencies and amplitudes of P1 and N1 for each modality separately for all phases of the experiment and correct the data of each condition relative to its pre-conditioing counterpart during the test phase, we do not believe that the pattern of results we report in the test phase is affected by the difference in visual and auditory reward cues intensity. This is however a possibility that could be explored in future studies. We have now added these explanations to line 738-746 of the revised manuscript: “In this vein, our results also indicated that the perceived intensity of visual and auditory stimuli might have been different as both P1 and N1 components were stronger for auditory compared to visual stimuli during the conditioning phase. Since we quantified the ERP measures separately for each sensory modality, calculated the reward effects against stimuli from the same modality, and corrected for pre-conditioning biases, it is unlikely that our reported results are due to the differences in perceived intensity of auditory and visual stimuli. We note however that using an adaptive method to equalize the perceived intensity of auditory and visual stimuli would be an interesting addition to future studies.” 3- During the conditioning phase, is there a difference between the first and the second half of the trials? If participants learn the association between a particular stimulus and either high or low reward, a difference between the two migh emerge later on during the experimental session. This is a great point. In order to test whether our results during the conditioning (reaction times, P3 amplitude and N1 latencies) are affected by the learning of reward associations which evolves in time, we repeated each analysis by including the phase of learning (i.e., first or second half of conditioning) as a factor. We did not observe a significant interaction between any of the reported effects and the phase of conditioning (first versus second half). These results (below) are now presented in lines 448-457 of the revised manuscript: “Since learning of reward associations may take time and behavioral and ERP effects of reward could only arise after associative learning has been completed, we wondered whether our reported results differed between the first and the second half of the conditioning. To test this possibility, we divided the trials for each condition to two halves and entered an extra factor, i.e., phase (first or second half of conditioning) in all our ANOVAs. We found similar effect sizes in each case (the interaction between reward and modality for RT: F(1,35) = 7.70, p = 0.009, ηp2= 0.18 and for N1 latency: F(1,35) = 7.26, p = 0.011, ηp2= 0.172, and the main effect of reward on P3 amplitude: F(1,35) = 4.27, p = 0.046, ηp2 = 0.109), but we did not observe a significant interaction with the phase (all ps0.1). Therefore, our reported results did not show a dependence on the phase of conditioning, likely because full learning of reward associations was achieved very fast.” 4-Figure 3e-f and Figure 5c: individual subjects data could be overlayed on top of the bar plots so to better appreciate the interindividual variability of the effect. Thanks for this suggestion. We have now added the individual data to both figures (3e-f and 5c). line 124: Figure 4d actually refers to Figure 2b, please correct the typo. Thank you for brining this error to our attention (it was in line 424). We have now corrected the typo (from Figure 4d to Figure 2b). 1. King AJ. Multisensory Integration: Strategies for Synchronization. Curr Biol. 2005;15: R339–R341. doi:https://doi.org/10.1016/j.cub.2005.04.022 2. Ragot R, Cave C, Fano M. Reciprocal effects of visual and auditory stimuli in a spatial compatibility situation. Bull Psychon Soc. 1988;26: 350–352. doi:10.3758/BF03337679 Attachment Submitted filename: RebuttalLetter_secondRevision.docx Click here for additional data file. 10.1371/journal.pone.0287900.r005 Decision Letter 2 Megna Nicola Academic Editor © 2023 Nicola Megna 2023 Nicola Megna https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version2 15 Jun 2023 Differential effects of intra-modal and cross-modal reward value on perception: ERP evidence PONE-D-22-27111R2 Dear Dr. Pooresmaeili, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Nicola Megna, M.D. Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: ==== Refs References 1 Schultz W. Neuronal reward and decision signals: From theories to data. Physiol Rev. 2015;95 : 853–951. doi: 10.1152/physrev.00023.2014 26109341 2 Berridge KC , Kringelbach ML . Affective neuroscience of pleasure: Reward in humans and animals. Psychopharmacology. Springer; 2008. pp. 457–480. doi: 10.1007/s00213-008-1099-6 3 Rangel A , Camerer C , Montague PR . A framework for studying the neurobiology of value-based decision making. Nat Rev Neurosci. 2008;9 : 545–556. doi: 10.1038/nrn2357 18545266 4 Schultz W. Multiple reward signals in the brain. Nat Rev. 2000;1 : 199–207. doi: 10.1038/35044563 11257908 5 Pessoa L , Engelmann JB . Embedding reward signals into perception and cognition. Frontiers in Neuroscience Frontiers Media SA; Sep, 2010 p. 17 . doi: 10.3389/fnins.2010.00017 20859524 6 Haber SN . Neuroanatomy of Reward: A View from the Ventral Striatum. Neurobiology of Sensation and Reward. CRC Press/Taylor & Francis; 2011. Available: http://www.ncbi.nlm.nih.gov/pubmed/22593898 7 Shuler MG , Bear MF . Reward timing in the primary visual cortex. Science. 2006;311 : 1606–1609. doi: 10.1126/science.1123513 16543459 8 Rutkowski RG , Weinberger NM . Encoding of learned importance of sound by magnitude of representational area in primary auditory cortex. Proc Natl Acad Sci U S A. 2005;102 : 13664–13669. doi: 10.1073/pnas.0506838102 16174754 9 Pleger B , Blankenburg F , Ruff CC , Driver J , Dolan RJ . Reward facilitates tactile judgments and modulates hemodynamic responses in human primary somatosensory cortex. J Neurosci. 2008;28 : 8161–8168. doi: 10.1523/JNEUROSCI.1093-08.2008 18701678 10 Weil RS , Furl N , Ruff CC , Symmonds M , Flandin G , Dolan RJ , et al . Rewarding feedback after correct visual discriminations has both general and specific influences on visual cortex. J Neurophysiol. 2010;104 : 1746–1757. doi: 10.1152/jn.00870.2009 20660419 11 Stanisor L , van der Togt C , Pennartz CMA , Roelfsema PR . A unified selection signal for attention and reward in primary visual cortex. Proc Natl Acad Sci U S A. 2013;110 : 9136–9141. doi: 10.1073/pnas.1300117110 23676276 12 Goltstein PM , Coffey EBJ , Roelfsema PR , Pennartz CMA . In vivo two-photon Ca2+ imaging reveals selective reward effects on stimulus-specific assemblies in mouse visual cortex. J Neurosci. 2013;33 : 11540–11555. doi: 10.1523/JNEUROSCI.1341-12.2013 23843524 13 Tankelevitch L , Spaak E , Rushworth MFS , Stokes MG . Previously Reward-Associated Stimuli Capture Spatial Attention in the Absence of Changes in the Corresponding Sensory Representations as Measured with MEG. J Neurosci. 2020;40 : 5033–5050. doi: 10.1523/JNEUROSCI.1172-19.2020 32366722 14 Bayer M , Grass A , Schacht A . Associated valence impacts early visual processing of letter strings: Evidence from ERPs in a cross-modal learning paradigm. Cogn Affect Behav Neurosci. 2019;19 : 98–108. doi: 10.3758/s13415-018-00647-2 30341624 15 Hammerschmidt W , Kagan I , Kulke L , Schacht A . Implicit reward associations impact face processing: Time-resolved evidence from event-related brain potentials and pupil dilations. Neuroimage. 2018;179 : 557–569. doi: 10.1016/j.neuroimage.2018.06.055 29940283 16 Schacht A , Adler N , Chen P , Guo T , Sommer W . Association with positive outcome induces early effects in event-related brain potentials. Biol Psychol. 2012;89 : 130–136. doi: 10.1016/j.biopsycho.2011.10.001 22027086 17 Luque D , Beesley T , Morris RW , Jack BN , Griffiths O , Whitford TJ , et al . Goal-directed and habit-like modulations of stimulus processing during reinforcement learning. J Neurosci. 2017;37 : 3009–3017. doi: 10.1523/JNEUROSCI.3205-16.2017 28193692 18 Rossi V , Vanlessen N , Bayer M , Grass A , Pourtois G , Schacht A . Motivational salience modulates early visual cortex responses across task sets. J Cogn Neurosci. 2017;29 : 968–979. doi: 10.1162/jocn_a_01093 28129056 19 Bayer M , Rossi V , Vanlessen N , Grass A , Schacht A , Pourtois G . Independent effects of motivation and spatial attention in the human visual cortex. Soc Cogn Affect Neurosci. 2017;12 : 146–156. doi: 10.1093/scan/nsw162 28031455 20 Leo F , Noppeney U . Conditioned sounds enhance visual processing. Harris J , editor. PLoS One. 2014;9 : e106860. doi: 10.1371/journal.pone.0106860 25192387 21 Pooresmaeili A , FitzGerald THB , Bach DR , Toelch U , Ostendorf F , Dolan RJ . Cross-modal effects of value on perceptual acuity and stimulus encoding. Proc Natl Acad Sci U S A. 2014;111 : 15244–15249. doi: 10.1073/pnas.1408873111 25288729 22 Anderson BA . Value-driven attentional capture in the auditory domain. Attention, Perception, Psychophys. 2016;78 : 242–250. doi: 10.3758/s13414-015-1001-7 26494380 23 Kang G , Chang W , Wang L , Wei P , Zhou X . Reward enhances cross-modal conflict control in object categorization: Electrophysiological evidence. Psychophysiology. 2018;55 . doi: 10.1111/psyp.13214 30129668 24 Kang G , Wang L , Zhou X . Reward interacts with modality shift to reduce cross-modal conflict. J Vis. 2017;17 . doi: 10.1167/17.1.19 28114493 25 Sanz LRD , Vuilleumier P , Bourgeois A . Cross-modal integration during value-driven attentional capture. Neuropsychologia. 2018;120 : 105–112. doi: 10.1016/j.neuropsychologia.2018.10.014 30342964 26 Pessoa L. Multiple influences of reward on perception and attention. Vis cogn. 2015;23 : 272–290. doi: 10.1080/13506285.2014.974729 26190929 27 Chelazzi L , Perlato A , Santandrea E , Della Libera C . Rewards teach visual selective attention. Vision Res. 2013;85 : 58–72. doi: 10.1016/j.visres.2012.12.005 23262054 28 Failing M , Theeuwes J . Selection history: How reward modulates selectivity of visual attention. Psychonomic Bulletin and Review Springer New York LLC; Apr 1, 2018 pp. 514–538. doi: 10.3758/s13423-017-1380-y 29 Bourgeois A , Chelazzi L , Vuilleumier P . How motivation and reward learning modulate selective attention. Progress in Brain Research. Elsevier B.V.; 2016. pp. 325–342. doi: 10.1016/bs.pbr.2016.06.004 30 Anderson BA , Laurent PA , Yantis S . Value-driven attentional capture. Proc Natl Acad Sci U S A. 2011;108 : 10367–10371. doi: 10.1073/pnas.1104047108 21646524 31 Kiss M , Driver J , Eimer M . Reward Priority of Visual Target Singletons Modulates Event-Related Potential Signatures of Attentional Selection. Psychol Sci. 2009;20 : 245–251. doi: 10.1111/j.1467-9280.2009.02281.x 19175756 32 Sawaki R , Luck SJ , Raymond JE . How attention changes in response to incentives. J Cogn Neurosci. 2015;27 : 2229–2239. doi: 10.1162/jocn_a_00847 26151604 33 Van Den Berg B , Krebs RM , Lorist MM , Woldorff MG . Utilization of reward-prospect enhances preparatory attention and reduces stimulus conflict. Cogn Affect Behav Neurosci. 2014;14 : 561–577. doi: 10.3758/s13415-014-0281-z 24820263 34 Grahek I , Schettino A , Koster EHW , Andersen SK . Dynamic interplay between reward and voluntary attention determines stimulus processing in visual cortex. J Cogn Neurosci. 2021;33 : 2357–2371. doi: 10.1162/jocn_a_01762 34272951 35 Etzel JA , Cole MW , Zacks JM , Kay KN , Braver TS . Reward Motivation Enhances Task Coding in Frontoparietal Cortex. Cereb Cortex. 2016;26 : 1647–1659. doi: 10.1093/cercor/bhu327 25601237 36 Pessoa L. How do emotion and motivation direct executive control? Trends Cogn Sci. 2009;13 : 160–166. doi: 10.1016/j.tics.2009.01.006 19285913 37 Kristjánsson Á , Sigurjónsdóttir Ó , Driver J . Fortune and reversals of fortune in visual search: Reward contingencies for pop-out targets affect search efficiency and target repetition effects. Attention, Perception, Psychophys. 2010;72 : 1229–1236. doi: 10.3758/APP.72.5.1229 20601703 38 Della Libera C , Chelazzi L . Learning to Attend and to Ignore Is a Matter of Gains and Losses. Psychol Sci. 2009;20 : 778–784. doi: 10.1111/j.1467-9280.2009.02360.x 19422618 39 Anderson , Laurent , Yantis . Reward predictions bias attentional selection. Front Hum Neurosci. 2013;7 : 1–6. doi: 10.3389/fnhum.2013.00262 23355817 40 Calvert GA , Thesen T . Multisensory integration: Methodological approaches and emerging principles in the human brain. J Physiol Paris. 2004;98 : 191–205. doi: 10.1016/j.jphysparis.2004.03.018 15477032 41 Navarra J , Alsius A , Soto-Faraco S , Spence C . Assessing the role of attention in the audiovisual integration of speech. Inf Fusion. 2010;11 : 4–11. doi: 10.1016/j.inffus.2009.04.001 42 Talsma D , Doty TJ , Woldorff MG . Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration? Cereb Cortex. 2007;17 : 679–690. doi: 10.1093/cercor/bhk016 16707740 43 Alsius A , Navarra J , Campbell R , Soto-Faraco S . Audiovisual integration of speech falters under high attention demands. Curr Biol. 2005;15 : 839–843. doi: 10.1016/j.cub.2005.03.046 15886102 44 Cheng FPH , Saglam A , André S , Pooresmaeili A . Cross-modal integration of reward value during oculomotor planning. eNeuro. 2020;7 . doi: 10.1523/ENEURO.0381-19.2020 31996392 45 Bean NL , Stein BE , Rowland BA . Stimulus value gates multisensory integration. Eur J Neurosci. 2021 [cited 27 Mar 2021]. doi: 10.1111/ejn.15167 33667027 46 Turatto M , Mazza V , Umiltà C . Crossmodal object-based attention: Auditory objects affect visual processing. Cognition. 2005;96 . doi: 10.1016/j.cognition.2004.12.001 15925569 47 Egly R , Driver J , Rafal RD . Shifting Visual Attention Between Objects and Locations: Evidence From Normal and Parietal Lesion Subjects. J Exp Psychol Gen. 1994;123 : 161–177. doi: 10.1037//0096-3445.123.2.161 8014611 48 Teder-Sälejärvi WA , McDonald JJ , Di Russo F , Hillyard SA . An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Cognitive Brain Research. Brain Res Cogn Brain Res; 2002. pp. 106–114. doi: 10.1016/s0926-6410(02)00065-4 12063134 49 Watson P , Pearson D , Most SB , Theeuwes J , Wiers RW , Le Pelley ME . Attentional capture by Pavlovian reward-signalling distractors in visual search persists when rewards are removed. PLoS One. 2019;14 . doi: 10.1371/journal.pone.0226284 31830126 50 Failing MF , Theeuwes J . Nonspatial attentional capture by previously rewarded scene semantics. Vis cogn. 2015;23 . doi: 10.1080/13506285.2014.990546 51 Rutherford HJV , O’Brien JL , Raymond JE . Value associations of irrelevant stimuli modify rapid visual orienting. Psychon Bull Rev. 2010;17 : 536–542. doi: 10.3758/PBR.17.4.536 20702874 52 Anderson BA , Yantis S . Persistence of value-driven attentional capture. J Exp Psychol Hum Percept Perform. 2013;39 : 6–9. doi: 10.1037/a0030860 23181684 53 Mine C , Saiki J . Task-irrelevant stimulus-reward association induces value-driven attentional capture. Attention, Perception, Psychophys. 2015;77 : 1896–1907. doi: 10.3758/s13414-015-0894-5 25893470 54 Maclean MH , Giesbrecht B . Neural evidence reveals the rapid effects of reward history on selective attention. Brain Res. 2015;1606 : 86–94. doi: 10.1016/j.brainres.2015.02.016 25701717 55 Serences JT . Value-based modulations in human visual cortex. Neuron. 2008;60 : 1169–1181. doi: 10.1016/j.neuron.2008.10.051 19109919 56 Serences JT , Saproo S . Population response profiles in early visual cortex are biased in favor of more valuable stimuli. J Neurophysiol. 2010;104 : 76–87. doi: 10.1152/jn.01090.2009 20410360 57 Faul F , Erdfelder E , Lang AG , Buchner A . G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods. Psychonomic Society Inc.; 2007. pp. 175–191. doi: 10.3758/BF03193146 58 Brainard DH . The Psychophysics Toolbox. Spat Vis. 1997;10 : 433–436. doi: 10.1163/156856897X00357 9176952 59 Watson AB , Pelli DG . Quest: A Bayesian adaptive psychometric method. Percept Psychophys. 1983;33 : 113–120. doi: 10.3758/bf03202828 6844102 60 Macmillan NA , Creelman CD . Detection theory: A user’s guide. Detection theory: A user’s guide. New York, NY, US: Cambridge University Press; 1991. 61 Leo F , Romei V , Freeman E , Ladavas E , Driver J . Looming sounds enhance orientation sensitivity for visual stimuli on the same side as such sounds. Exp Brain Res. 2011;213 : 193–201. doi: 10.1007/s00221-011-2742-8 21643714 62 Delorme A , Makeig S . EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods. 2004. doi: 10.1016/j.jneumeth.2003.10.009 15102499 63 Mognon A , Jovicich J , Bruzzone L , Buiatti M . ADJUST: An automatic EEG artifact detector based on the joint use of spatial and temporal features. Psychophysiology. 2011. doi: 10.1111/j.1469-8986.2010.01061.x 20636297 64 Giard MH , Peronnet F . Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. J Cogn Neurosci. 1999;11 : 473–490. doi: 10.1162/089892999563544 10511637 65 Teder-Sälejärvi WA , Münte TF , Sperlich FJ , Hillyard SA . Intra-modal and cross-modal spatial attention to auditory and visual stimuli. An event-related brain potential study. Cogn Brain Res. 1999;8 : 327–343. doi: 10.1016/s0926-6410(99)00037-3 10556609 66 Talsma D , Woldorff MG . Selective attention and multisensory integration: Multiple phases of effects on the evoked brain activity. J Cogn Neurosci. 2005;17 : 1098–1114. doi: 10.1162/0898929054475172 16102239 67 Mangun GR . Neural mechanisms of visual selective attention. Psychophysiology. 1995;32 : 4–18. doi: 10.1111/j.1469-8986.1995.tb03400.x 7878167 68 Luck SJ , Hillyard SA , Mouloua M , Woldorff MG , et al . Effects of spatial cuing on luminance detectability: Psychophysical and electrophysiological evidence for early selection. J Exp Psychol Hum Percept Perform. 1994;20 : 887–904. doi: 10.1037//0096-1523.20.4.887 8083642 69 Woldorff MG , Fox PT , Matzke M , Lancaster JL , Veeraswamy S , Zamarripa F , et al . Retinotopic organization of early visual spatial attention effects as revealed by PET and ERPs. Human Brain Mapping. Hum Brain Mapp; 1997. pp. 280–286. doi: 10.1002/(SICI)1097-0193(1997)5:4<280::AID-HBM13>3.0.CO;2-I 20408229 70 Baines S , Ruz M , Rao A , Denison R , Nobre AC . Modulation of neural activity by motivational and spatial biases. Neuropsychologia. 2011;49 : 2489–2497. doi: 10.1016/j.neuropsychologia.2011.04.029 21570417 71 Lakens D . Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013;4 : 863. doi: 10.3389/fpsyg.2013.00863 24324449 72 Pick HL , Warren DH , Hay JC . Sensory conflict in judgments of spatial direction. Percept Psychophys. 1969;6 : 203–205. doi: 10.3758/BF03207017 73 Bertelson P , Radeau M . Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys. 1981;29 : 578–584. doi: 10.3758/bf03207374 7279586 74 Luque D , Morís J , Rushby JA , Le Pelley ME . Goal-directed EEG activity evoked by discriminative stimuli in reinforcement learning. Psychophysiology. 2015;52 : 238–248. doi: 10.1111/psyp.12302 25098203 75 Hickey C , Chelazzi L , Theeuwes J . Reward changes salience in human vision via the anterior cingulate. J Neurosci. 2010;30 : 11096–11103. doi: 10.1523/JNEUROSCI.1026-10.2010 20720117 76 McDonald JJ , Störmer VS , Martinez A , Feng W , Hillyard SA . Salient Sounds Activate Human Visual Cortex Automatically. J Neurosci. 2013;33 : 9194. Available: http://www.jneurosci.org/content/33/21/9194.abstract doi: 10.1523/JNEUROSCI.5902-12.2013 23699530 77 Feng W , Stormer VS , Martinez A , McDonald JJ , Hillyard SA . Sounds Activate Visual Cortex and Improve Visual Discrimination. J Neurosci. 2014. doi: 10.1523/JNEUROSCI.4869-13.2014 25031419 78 Weinberger NM . Associative representational plasticity in the auditory cortex: A synthesis of two disciplines. Learning and Memory. Learn Mem; 2007. pp. 1–16. doi: 10.1101/lm.421807 17202426 79 Wikman P , Rinne T , Petkov CI . Reward cues readily direct monkeys’ auditory performance resulting in broad auditory cortex modulation and interaction with sites along cholinergic and dopaminergic pathways. Sci Rep. 2019;9 . doi: 10.1038/s41598-019-38833-y 30816142 80 Goldstein RZ , Cottone LA , Jia Z , Maloney T , Volkow ND , Squires NK . The effect of graded monetary reward on cognitive event-related potentials and behavior in young healthy adults. Int J Psychophysiol. 2006;62 : 272–279. doi: 10.1016/j.ijpsycho.2006.05.006 16876894 81 Pornpattananangkul N , Nusslock R . Motivated to win: Relationship between anticipatory and outcome reward-related neural activity. Brain Cogn. 2015;100 : 21–40. doi: 10.1016/j.bandc.2015.09.002 26433773 82 Yeung N , Sanfey AG . Independent coding of reward magnitude and valence in the human brain. J Neurosci. 2004;24 : 6258–6264. doi: 10.1523/JNEUROSCI.4537-03.2004 15254080 83 Störmer VS , McDonald JJ , Hillyard SA . Cross-modal cueing of attention alters appearance and early cortical processing of visual stimuli. Proc Natl Acad Sci U S A. 2009;106 : 22456–22461. doi: 10.1073/pnas.0907573106 20007778 84 Busse L , Roberts KC , Crist RE , Weissman DH , Woldorff MG . The spread of attention across modalities and space in a multisensory object. Proc Natl Acad Sci U S A. 2005;102 : 18751–18756. doi: 10.1073/pnas.0507704102 16339900 85 Zimmer U , Itthipanyanan S , Grent-’T-Jong T , Woldorff MG . The electrophysiological time course of the interaction of stimulus conflict and the multisensory spread of attention. Eur J Neurosci. 2010;31 : 1744–1754. doi: 10.1111/j.1460-9568.2010.07229.x 20584178 86 Molholm S , Ritter W , Murray MM , Javitt DC , Schroeder CE , Foxe JJ . Multisensory auditory-visual interactions during early sensory processing in humans: A high-density electrical mapping study. Cognitive Brain Research. Elsevier; 2002. pp. 115–128. doi: 10.1016/S0926-6410(02)00066-6 87 Teder-Sälejärvi WA , Di Russo F , McDonald JJ , Hillyard SA . Effects of spatial congruity on audio-visual multimodal integration. J Cogn Neurosci. 2005;17 : 1396–1409. doi: 10.1162/0898929054985383 16197693 88 Watson R , Latinus M , Noguchi T , Garrod O , Crabbe F , Belin P . Crossmodal adaptation in right posterior superior temporal sulcus during face-voice emotional integration. J Neurosci. 2014;34 : 6813–6821. doi: 10.1523/JNEUROSCI.4478-13.2014 24828635 89 Franzen L , Delis I , De Sousa G , Kayser C , Philiastides MG . Auditory information enhances post-sensory visual evidence during rapid multisensory decision-making. Nat Commun. 2020;11 : 1–14. doi: 10.1038/s41467-020-19306-7 31911652 90 Pooresmaeili A , Roelfsema PR . A growth-cone model for the spread of object-based attention during contour grouping. Curr Biol. 2014;24 . doi: 10.1016/j.cub.2014.10.007 25456446 91 Feldmann-Wüstefeld T , Schubö A . Context homogeneity facilitates both distractor inhibition and target enhancement. J Vis. 2013;13 . doi: 10.1167/13.3.11 23650629 92 Chang S , Egeth HE . Can salient stimuli really be suppressed? Atten Percept Psychophys. 2021;83 : 260–269. doi: 10.3758/s13414-020-02207-8 33241528 93 Anderson BA , Laurent PA , Yantis S . Learned value magnifies salience-based attentional capture. PLoS One. 2011;6 . doi: 10.1371/journal.pone.0027926 22132170 94 Gong M , Yang F , Li S . Reward association facilitates distractor suppression in human visual search. Foxe J , editor. Eur J Neurosci. 2016;43 : 942–953. doi: 10.1111/ejn.13174 26797805 95 Anderson BA . A value-driven mechanism of attentional selection. J Vis. 2013;13 . doi: 10.1167/13.3.7 23589803 96 Rusz D , Le Pelley ME , Kompier MAJ , Mait L , Bijleveld E . Reward-Driven distraction: A meta-analysis. Psychol Bull. 2020;146 : 872–899. doi: 10.1037/bul0000296 32686948 97 Rusz D , Bijleveld E , Kompier MAJ . Reward-associated distractors can harm cognitive performance. PLoS One. 2018;13 . doi: 10.1371/journal.pone.0205091 30286146 98 Yantis S , Anderson BA , Wampler EK , Laurent PA . Reward and Attentional Control in Visual Search. Nebraska Symposium on Motivation Nebraska Symposium on Motivation. Nebr Symp Motiv; 2012. pp. 91–116. doi: 10.1007/978-1-4614-4794-8_5 23437631 99 Hickey C , Di Lollo V , McDonald JJ . Electrophysiological indices of target and distractor processing in visual search. J Cogn Neurosci. 2009;21 : 760–775. doi: 10.1162/jocn.2009.21039 18564048 100 Krebs RM , Boehler CN , Egner T , Woldorff MG . The neural underpinnings of how reward associations can both guide and misguide attention. J Neurosci. 2011;31 : 9752–9759. doi: 10.1523/JNEUROSCI.0732-11.2011 21715640 101 Krebs RM , Boehler CN , Appelbaum LG , Woldorff MG . Reward Associations Reduce Behavioral Interference by Changing the Temporal Dynamics of Conflict Processing. PLoS One. 2013;8 . doi: 10.1371/journal.pone.0053894 23326530 102 Bustamante L , Lieder F , Musslick S , Shenhav A , Cohen J . Learning to Overexert Cognitive Control in a Stroop Task. Cogn Affect Behav Neurosci. 2021 [cited 24 Mar 2021]. doi: 10.3758/s13415-020-00845-x 33409959 103 Krebs RM , Boehler CN , Woldorff MG . The influence of reward associations on conflict processing in the Stroop task. Cognition. 2010;117 : 341–347. doi: 10.1016/j.cognition.2010.08.018 20864094 104 Grégoire L , Anderson BA . Semantic generalization of value-based attentional priority. Learn Mem. 2019;26 : 460–464. doi: 10.1101/lm.050336.119 31732706 105 Itthipuripat S , Vo VA , Sprague TC , Serences JT . Value-driven attentional capture enhances distractor representations in early visual cortex. PLoS Biol. 2019;17 : e3000186. doi: 10.1371/journal.pbio.3000186 31398186 106 Hopf JM , Boehler CN , Luck SJ , Tsotsos JK , Heinze HJ , Schoenfeld MA . Direct neurophysiological evidence for spatial suppression surrounding the focus of attention in vision. Proc Natl Acad Sci U S A. 2006;103 : 1053–1058. doi: 10.1073/pnas.0507746103 16410356 107 Mounts JRW . Attentional capture by abrupt onsets and feature singletons produces inhibitory surrounds. Percept Psychophys. 2000;62 : 1485–1493. doi: 10.3758/bf03212148 11143458 108 Störmer VS , Alvarez GA . Feature-based attention elicits surround suppression in feature space. Curr Biol. 2014;24 : 1985–1988. doi: 10.1016/j.cub.2014.07.030 25155510 109 Engelmann JB . Combined effects of attention and motivation on visual task performance: Transient and sustained motivational effects. Front Hum Neurosci. 2009;3 : 4. doi: 10.3389/neuro.09.004.2009 19434242 110 Nobre AC , Rao A , Chelazzi L . Selective attention to specific features within objects: Behavioral and electrophysiological evidence. J Cogn Neurosci. 2006;18 : 539–561. doi: 10.1162/jocn.2006.18.4.539 16768359 111 Ragot R , Cave C , Fano M . Reciprocal effects of visual and auditory stimuli in a spatial compatibility situation. Bull Psychon Soc. 1988;26 : 350–352. doi: 10.3758/BF03337679