
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

39294213
71618
10.1038/s41598-024-71618-6
Article
Assessment of verbal memory in Parkinson's disease utilizing a virtual reality-based Rey Auditory Verbal Learning Test
Gottlieb Amihai amihai.gottlieb@gmail.com

1
Kimel-Naor Shani 1
Zeilig Gabi 234
Schnaider Beeri Michal 5
Plotnik Meir 167
1 https://ror.org/020rzx487 grid.413795.d 0000 0001 2107 2845 Center of Advanced Technologies in Rehabilitation, Sheba Medical Center, Tel Hashomer, Ramat Gan, Israel
2 https://ror.org/020rzx487 grid.413795.d 0000 0001 2107 2845 Department of Neurological Rehabilitation, Sheba Medical Center, Tel Hashomer, Ramat Gan, Israel
3 https://ror.org/04mhzgx49 grid.12136.37 0000 0004 1937 0546 Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
4 https://ror.org/02td5wn81 grid.430101.7 0000 0004 0631 5599 School of Health Professions, Ono Academic College, Kiryat Ono, Israel
5 https://ror.org/029z02k15 The Hebert and Jacqueline Krieger Klein Alzheimer’s Research Center, Brain Health Institute, Rutgers Health, Newark, NJ USA
6 https://ror.org/04mhzgx49 grid.12136.37 0000 0004 1937 0546 Department of Physiology and Pharmacology, Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
7 https://ror.org/04mhzgx49 grid.12136.37 0000 0004 1937 0546 Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
18 9 2024
18 9 2024
2024
14 2179228 2 2024
29 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
The Rey Auditory Verbal Learning Test (RAVLT) is a commonly used tool for evaluating verbal learning and memory in neuropsychological assessments. In recent years, we developed a Virtual Reality (VR) adaptation of the RAVLT (VR-RVLT), aiming for increased ecological validity compared to the traditional pen and paper gold standard (GS-RAVLT). Following validation in healthy cohorts, the VR-RAVLT was validated with thirty individuals with Parkinson's Disease (PD) that completed both the GS-RAVLT and the VR- RAVLT. Validity of the VR-RAVLT was evaluated by assessing its construct and discriminant validity, and test–retest reliability, in comparison to the GS-RAVLT. Results of the PD participants were compared to those of 46 previously recruited healthy participants with comparable age and level of education. Main outcome measures derived from the remembered items on the test lists, exhibited significant and comparable correlations between VR-RAVLT and GS-RAVLT, both among healthy participants and PD participants. Likewise, serial position effects were similar for both formats amog the PD participants. Additionally, both formats showed similar discriminatory ability between healthy controls and PD participants, as well as comparable test–retest reliability measures. Taken together, the results suggest that the VR-based RAVLT is equally effective in measuring verbal memory capabilities in individuals with PD as compared to the GS-RAVLT. Certain results indicate that the virtual reality version has the capability to encompass additional factors that might impact memory performance, thereby suggesting an enhanced ecological validity.

Keywords

Memory and learning tests
Neuropsychological tests
Parkinson's disease
Rey Auditory Verbal Learning Test
Virtual reality
Subject terms

Parkinson's disease
Learning and memory
Cognitive neuroscience
Dr. Marrina Nissim Ms. Judi Steiissue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Memory tests are among the most widely employed neuropsychological assessments for identifying potential cognitive impairment and associated diseases. These tests are particularly significant in the context of neurodegenerative diseases, as they serve an integral tool in the diagnostic process and for monitoring progressive decline and treatment efficacy1. In Parkinson's Disease (PD), a progressive neurodegenerative condition affecting more than 8.5 million people globally2, 78% of patients ultimately develop memory impairment and dementia3.

The Rey Auditory Verbal Learning Test (RAVLT)4,5 is a standardized memory assessment used to evaluate an individual's verbal learning and memory abilities. Briefly, the examinee is required, several times, to recite list of fifteen items read to him/her and the scoring is based on the number of items that were recited (the more, the better). The RAVLT is widely used in various populations, including healthy individuals and clinical cohorts1. Its effectiveness is of particular clinical value in the assessment of neurodegenerative diseases as different dementias exhibit different patterns of response and it is a strong predictor of cognitive decline even in the very early stages1. The RAVLT is specifically recognized as a tool for differentiating between types of dementia caused by various diseases1, such as Alzheimer's disease (AD) and Parkinson's disease (PD), due to their distinct result patterns. For example, patients with AD typically show a more pronounced recency effect compared to the primacy effect during the learning phase of Trials 1–5, unlike control groups and patients with PD6,7.

In clinical practice the RAVLT is administered using traditional pen and paper (P&P) settings. In recent years, with the advancements of the virtual reality (VR) technology, a new opportunity has risen, that is, to adapt the traditional P&P tests for administration using VR technology. VR adaptations have been proposed as a solution to a significant challenge posed by traditional P&P neuropsychological tests—namely, their lack of ecological validity, manifested as a disconnect between test performance and real-life functioning8,9. VR technology has the potential to address this issue by better mimicking daily life scenarios while preserving the psychometric integrity of the original tests7,8. Moreover, VR adaptations of neuropsychological tests can capture functions more closely related to daily life, potentially offering a superior solution for detecting mild cognitive decline compared to traditional tests. This is a critical need in conditions like PD, where there is compelling evidence indicating the urgent necessity for sensitive test able to identify subtle changes in cognitive function before PD onset10,11.

We have recently successfully developed and validated a VR adaptation of the RAVLT (VR-RAVLT)12. Within the VR adaptation, the examinee is participating in a conversation with a secretary in a virtual office. To maintain the core features of the original Gold Standard RAVLT (GS-RAVLT), the virtual secretary lists places that the examinee needs to visit that day, and s/he is required to repeat those places. The new VR adaptation was validated with the participation of healthy young- middle- and older- adults. Construct validity, discriminant validity, and test–retest reliability were demonstrated using GS-RAVLT for comparisons12. In this study, we aimed at broadening the validation of the VR-RAVLT to PD.

The current study's objective was to assess the construct and discriminant validities and the test–retest reliability of the VR-RAVLT in comparison to the GS-RAVLT specifically among persons with Parkinson's Disease (PwPD). For this aim, we paralleled the validation process used for healthy individuals12 and compared the results of the PwPD group with those of previously recruited healthy controls. To ensure consistency and enable meaningful comparisons, we closely adhered to the protocol outlined in our previous paper12 while working with the newly recruited clinical cohort.

Material and methods

Participants

A sample of 30 PwPD was recruited for the study. Recruitment was conducted through a convenience sampling method. Clinicians from the rehabilitation hospital identified suitable persons who were diagnosed with PD, based on medical records. Those persons were contacted and offered to participate in the study. In addition, the results of 49 previously recruited healthy controls were also included in the analysis. For the PwPD cohort, the inclusion criteria consisted of the following: Normal cognitive function as determined by the 'Montreal Cognitive Assessment' (MoCA), visual acuity of 20/20 (or corrected to 20/20), fluency in Hebrew, and a confirmed diagnosis of PD. Normal cognitive function was determined using a MoCA with a cutoff score of 24, whereas scores between 24 and 26 indicate mild cognitive impairment13. Exclusion criteria were having any motor, balance, psychiatric, or cognitive impairment that would interfere with understanding instructions or performing the tasks. Only one PwPD participant (male, age 61) was excluded from the study due to inability to use the VR goggles. All participants signed a written informed consent prior to entering the study. The study adhered to the principles of the Helsinki Declaration and obtained approval from the local Sheba Medical Center Institutional Review Board committee, under Approval Number 3524-16-SMC. Table 1 presents the demographic details of all participants and the clinical data specific to the PwPD participants. Table 1 Demographic and clinical data.

	PwPD (n = 30)	Healthy controls (n = 49)	P (T-Test/Chi square)	
Age (mean ± SD, range)	64.3 ± 7.2 (48–75)	63.9 ± 7.9 (52–86)	p = 0.83	
Gender (females, percentage)	10, 33.3%	34, 69.4%	p < 0.001*	
Years of education (mean ± SD, range)	15.1 ± 2.5 (11–21)	15.6 ± 3.8 (8–30)	p = 0.55	
MoCA score (mean ± SD, range)	25.06 ± 2.8 (19–30)	26.1 ± 2.2 (20–30)	p = 0.07	
Duration of PD (mean ± SD, range)	7.1 ± 5.1 (1–21)	N/A	N/A	
H&Y score**	2.21 ± 0.7	N/A	N/A	
ANOVA Analysis Of Variance, H&Y Hoehn and Yahr Scale14, MoCA Montreal Cognitive Assessment, PD Parkinson's Disease PwPD People With Parkinson's Disease, SD Standard Deviation.

*Chi square, **H&Y score was recorded for 27 PwPD during ON phase and for 3 PwPD during OFF phase.

Sample size justification

To determine the necessary sample size of the PwPD participants, we utilized the Acquisition measure (see below) as the main outcome measure extracted from the results of a prior study by Broeders et al.15, in which 59 PwPD and 40 healthy controls completed the GS-RAVLT. The mean Acquisition values (± SD) were 41.4 ± 9.6 and 50.7 ± 7.7 for the PwPD and the healthy control group, respectively. Using these values and Gpower16, we found that in order to demonstrate a significant difference between the groups (alpha = 2.5%) with 90% power we need a minimum of 24 participants for each group. However, given that one of the aims of the current study was to compare test–retest reliability and considering high attrition rates reported among PD patients17, we planned to recruit 30 PwPD participants to account for an expected 25% rate of drop-out.

Apparatus and virtual environment

Full description of the VR system and of the VR environment (i.e., VR-RAVLT) was provided in Gottlieb et al.12. Briefly, the virtual adaptation of the RAVLT was created as a virtual office assistant asking the participants to memorize a list of 15 places to visit on the next day as it will not be able to assist him/her while later asking them to repeat the list according to the original RAVLT's procedure. The test was done using a full immersive VR system (HTC-Vive; New Taipei City, Taiwan).

Procedure

The RAVLT entails the repetition of a 15-word list, which the individual is tasked with remembering and repeating back in any sequence over five trials. Following the last trial, the individual is prompted to recall as many words as they can from the list without any assistance. Subsequently, a new list of unrelated words is introduced, and the individual is once again asked to recall the initial list. Lastly, after a lapse of 20–30 min, the participants are prompted to recall the original list once more. The test is scored based on the number of words correctly remembered across all trials and the degree of forgetting between trials4,5.

The GS-RAVLT was administered following the above instructions outlined in its handbook5. The VR-RAVLT was conducted following the established procedure detailed in the study by Gottlieb et al.12. The participants performed GS-RAVLT and the VR-RAVLT test in a full counter-balanced order. Similarly to the healthy control group12, PwPD also performed the VR-RAVLT while validating two alternate forms, i.e., half memorized one form and the other half memorized another form. However, since, in a pre-hoc analysis it was found that the two alternate forms produced comparable results for PwPD (as also previously reported for healthy participants12), we have combined the results from the two forms.

To assess test–retest reliability, the GS-RAVLT and the VR-RAVLT were administered to the participants in two separate visits, with a 2–3 weeks' interval. However, due to participant dropout, only 26 PwPD participants completed the RAVLTs during the second visit.

Outcome measures and analyses

The number of correct remembered items was recorded for both the GS-RAVLT and the VR-RAVLT, and summary statistics (mean + SD) were calculated. The primary study outcomes included Acquisition (the sum of the remembered items in the first five trials, the more the better) and Retention (the subtraction of the remembered items in the fifth trial from those in the delayed recall trial, the less the better). Due to limited usefulness of the RAVLT's Retroactive Interference measure identified in the previous validation study12, i.e., limited ability to differentiate between age groups and low test–retest reliability, it was excluded from the current study.

To determine the appropriateness of using parametric statistical methods for data analysis, we conducted Shapiro–Wilk normality tests that were run on the residuals of four analyses of variance (ANOVA) with a two-level independent group variable that were performed on both main outcome measures, per test. This analysis revealed that all four of the normality tests indicated non-normal distributions (Shapiro–Wilk statistic ≤ 0.9; p ≤ 0.01). Therefore, we used non-parametric Mann–Whitney u tests to examine the effects of Group (PwPD, controls) within each test format and Wilcoxon signed rank tests to examine the effects of Format (GS vs. VR), for both main outcome measures (Acquisition and Retention).

Construct validity of the VR-RAVLT was validated by conducting correlation analyses (Spearman) between the primary outcome measures obtained from each test format. We previously reported construct validity of the VR-RAVLT among healthy participants12, yet here we re-executed the same correlation analyses for the healthy control group to facilitate comparison. Furthermore, we assessed construct validity by examining the serial position effects within the PwPD group. These effects were previously demonstrated and reported in the context of the healthy control group12. For these effects, including primacy and recency (as described below), we tallied the occurrences of each word in the list (i.e., word #1, word #2, …, word #15) as they were recited. This cumulative count was obtained by combining responses from all PwPD participants. To determine the 'remembering proportion' for each word, we divided this count by the total number of times that the word could have been recited (seven times for each participant). To explore whether the well-documented Primary and Recency effects, i.e., which reflect the inclination to recall the first and last words read in a list18,19, is also manifest in PwPD when assessed with the VR-RAVLT, we divided these proportions into three segments: Primacy (words #1–5), Middle (words #6–10), and Recency (words #11–15). This segmentation aligns with established literature18,19. Finally, these scores were analyzed using a repeated measures ANOVA, considering the segments and test formats as within-subjects factors.

Discriminant validity of the VR-RAVLT (i.e., ability to separate between the PwPD and the healthy controls groups) was validated by computing the Receiver Operating Characteristic (ROC) Area Under the Curve values (AUC; range: 0–1, higher values reflect better discriminability) for the main outcome measures, and comparing them to the GS-RAVLT values.

Test–retest reliability of the VR-RAVLT was assessed by computing Intraclass Correlation Coefficients values (ICC; two-way mixed effects, absolute agreement) for the main outcome measures. The ICC values were obtained by comparing the results of the GS-RAVLT and the VR-RAVLT performed by the PwPD participant cohort in two visits (test–retest reliability among healthy controls was published before12), spaced 2–3 weeks apart. Good reliability is conventionally defined as ICC > 0.7520.

Results

In general, participants successfully completed the VR-RAVLT platform and reported no complaints regarding discomfort or difficulties (see also12). Only one PwPD participant was not able to complete the VR-RAVLT due to visual difficulties associated with the use of the VR apparatus.

RAVLT performance in PwPD and healthy control—effects of group and format

Table 2 presents the Summary statistics (mean ± SD) that were computed for each outcome measure. Table 2 Summary statistics for each outcome measure.

	VR-RAVLT (mean ± SD, range)	GS-RAVLT (mean ± SD, range)	
PwPD	Healthy Control	PwPD	Healthy Control	
Acquisition (0–75)	44.6 ± 11.0 (13–63)	50.8 ± 8.6 (25–64)*	43.9 ± 12.9 (10–69)	53.0 ± 11.8 (27–71)*	
Retention (− 15 to 15)†	1.8 ± 2.0 (− 2 to + 8)	1.2 ± 2.0 (− 3 to + 10)	3.3 ± 2.0 (0 to + 9)‡	1.6 ± 2.5 (− 2 to + 10)*	
ACQUISITION Sum of trial 1–5, PwPD people with Parkinson's disease, SD standard deviation, RETENTION trial 5—delayed recall.

*p ≤ 0.006 (group comparison); †p = .003 (format effect); ‡p = .003 (format effect within group); For more on statistics see text.

The VR-RAVLT demonstrated a group effect for the Acquisition variable, with poorer scores (less remembered items) observed for the PwPD group (H(2) = − 2.8, p = 0.005). For Acquisition, group effect was also signification for the GS-RAVLT (H(2) = − 2.7, p = 0.006). For the Retention parameter, group effect was significant only for the GS-RAVLT (H(2) = − 3.4, p = 0.001), but not for the VR-RAVLT (H(2) = − 1.3, p = 0.18).

Analysis of overall format effects revealed no significant format difference with regards to the Acquisition variable (Z = − 0.92, p = 0.35) and significant differences between the VR-RAVLT and the GS-RAVLT in reference to the Retention variable, with the VR-RAVLT exhibiting better performance (Z = 2.94, p = 0.003). However, when performing the format analyses within groups, the Retention variable was found to be significantly different between formats only for the PwPD group (Z = 3.0, p = 0.003) and not for the healthy control group (Z = 1.2, p = 0.19), whereas the Acquisition variable remained non-significant in both groups.

Construct validity

The analysis of Acquisition correlations obtained from both test formats demonstrated similarly high and statistically significant correlations among both PwPD (rs = 0.69, p < 0.0001) and healthy controls (rs = 0.61, p < 0.0001). Figure 1A portrays the relationship between the performance of both groups on the GS-RAVLT and the VR-RAVLT concerning the Acquisition outcome measure. Albeit less robust than for Acquisition, significant correlations for Retention were found both in PwPD (rs = 0.40, p = 0.02) and in Healthy controls (rs = 0.30, p = 0.03). Figure 1B depicts the relationship between the performance of both groups on the GS-RAVLT and the VR-RAVLT concerning the Retention outcome measure. Since Retention is relatively discrete variable (see Fig. 1B) and in order to explore correlation trends, we grouped the data based on the scores on the GS-RAVLT (Fig. 1C).Fig. 1 Correlation between scores on the VR-RAVLT and the GS-RAVLT. (A) Acquisition scores of the VR-RAVLT plotted against the scores on the GS-RAVLT for both study groups (see key). Dotted lines indicate linear fits (see equations). Diamonds and thick lines adjacent to the axes indicate mean values and standard deviations, respectively. (B) Retention values obtained using the VR-RAVLT plotted against the values recorded from the GS-RAVLT. Diamonds and bold lines close to the axes represent the averages and standard deviations. Given the discrete nature of the data, (C) illustrates the mean values of VR-RAVLT Retention scores across the GS-RAVLT scores. Dotted lines indicate linear fits (see equations). Error bars = standard deviations. In panels (B) and (C), to enhance visual clarity and prevent overlap between data points, a margin of 0.05 was added to the PwPD and subtracted from the Control's GS-RAVLT's data points ('x axis'). In Panel (C), PD VR-RAVLT mean Retention scores 1.8 ± 2.0. PD GS-RAVLT Retention scores—3.3 ± 2.0. Control VR-RAVLT Retention scores—1.2 ± 2.0. Control GS-RAVLT Retention scores 1.6 ± 2.5.

The analysis of the serial position effect showed a significant main effect of segments (F(2,12) = 13.2, p < 0.0001, η2 = 0.68), indicating a strong pattern of primacy and recency effects in both tests (Fig. 2). There was no significant main effect of test format (F(1,12) = 0.70, p = 0.41, η2 = 0.05), nor was there any interaction effect (F(2,12) = 0.016, p = 0.85, η2 = 0.02). Figure 2 provide a visual depiction of the serial position effects among the PwPD participants.Fig. 2 Cumulative serial position curves for the 15 words across five trials among the PwPD group for both the GS-RAVLT and the VR-RAVLT. The curves depict the proportions of remembered items as a function of the word's serial positions Data aggregated across participants. Clear similarities in Primacy (words 1–5) and Recency effects (words 11–15) can be observed.

Discriminant validity

Table 3 presents the AUC values extracted from the ROC curves for discriminating between the two cohorts. Overall, the comparability between the two formats is evident regarding Acquisition discriminability, but there is a noticeable difference in their Retention ability to discriminate between groups with superiority to the GS-RAVLT. Table 3 AUC values extracted from ROC curves.

	PwPD versus control	
GS-RAVLT	Acquisition	0.68*	
Retention	0.73**	
VR-RAVLT	Acquisition	0.68*	
Retention	0.58	
AUC area under the curve, PwPD people with Parkinson's disease, ROC receiver operating characteristic curves; Details on actual scores are not shown.

*p ≤ .05; **p < .0001.

Test–retest reliability

Table 4 presents the test–retest ICC values for both the PwPD and the control groups, categorized with reference to the test formats and main outcome measures. For the Acquisition measure, the statistically significant ICC values were comparable for both formats and for both groups. For the Retention variable, however, ICC value was not statistically significant only for the control group when using the VR-RAVT. Table 4 ICC values.

		PwPD	Control	
Acquisition	GS-RAVLT	ICC = 0.854, p < 0.0001	ICC = 0.879, p < 0.0001	
VR-RAVLT	ICC = 0.882, p < 0.0001	ICC = 0.755, p < 0.0001	
Retention	GS-RAVLT	ICC = 0.513, p = 0.02	ICC = 0.819, p < 0.0001	
VR-RAVLT	ICC = 0.669, p < 0.0001	ICC = 0.275, p = 0.14	
ICC intraclass correlation coefficients, PwPD people with Parkinson's disease.

Control analysis

Due to the gender disparity between the PwPD and the control group (see Table 1), and in order to control for the possibility that the significant group difference arises from this distinction (i.e., as it was reported that women tend to perform better in verbal memory than men16,17,20), we conducted a control analysis wherein we matched participants in both groups based on gender, age, and education, resulting in 10 females and 20 males in each group. Subsequently, we reran the aforementioned analyses. The results indicate that the group differences remained similar. The PwPD group showed poorer, nearly significant acquisition scores in the GS-RAVLT (H(2) =  − 1.8, p = 0.06) and significant results for the VR-RAVLT (H(2) = − 2.5, p = 0.01). For the retention parameter, the group effect remained significant only for the GS-RAVLT (H(2) = − 2.8, p = 0.004) but not for the VR-RAVLT (H(2) = − 1.7, p = 0.08). The other results remained relatively unchanged (data not shown).

Discussion

Following the validation of the VR-RAVLT in healthy adults across age groups12, the aim of this study was to expand its validation to PD. For this, a total of 30 PwPD completed both the VR-RAVLT and the GS-RAVLT, and their results were compared to those of a previously recruited age and education comparable healthy controls (n = 49).

Summary of findings

Construct validity was supported by significant correlations between VR-RAVLT and GS-RAVLT performance in PwPD and control groups for both Acquisition and Retention. In addition, Serial Position Effects in PwPD indicated comparable primacy and recency effects for both formats. Discriminant validity analyses using AUC values revealed both formats effectively discriminated between PwPD and healthy controls in the Acquisition variable. However, GS-RAVLT outperformed VR-RAVLT in discriminating cohorts for Retention due to reduced forgetfulness in the PwPD group with VR-RAVLT. Finally, Test–retest reliability indicated similar results for Acquisition in both formats while VR-RAVLT showed superior reliability for Retention.

Interpretation of results

Both the feasibility and usability aspects reported here and in our previous study12, and the current construct validity analyses, support that the VR-RAVLT maintains the core features of the GS-RAVTL and is a valid technology-based test for the assessment of verbal memory. Specifically, two main outcomes in verbal memory evaluation, i.e., Acquisition and Retention, were correlated between the traditional and the VR formats, and the serial position effects, support this conclusion.

As described, the GS-RAVLT format outperformed the VR-RAVLT format in discriminating between the cohorts when using the Retention variable, most likely due to the fact that the PwPD group demonstrated better memory in response to the VR-RAVLT compared to their own performance in the GS-RAVLT. Why only PwPD demonstrate better memory in VR as compared to the traditional test for the Retention variable and not in the Acquisition variable? And why a similar result was not seen for the healthy control group? One factor that may play a role is an enhanced encoding ability facilitated by the VR-based test as the saliency of the VR experience may have contributed to the decreased forgetfulness of the target words, as saliency has been shown to improve remembrance21,22. This effect was specific for the PwPD group and not observed in the healthy controls, potentially due to 'floor effect' among the healthy participants, i.e. forgetfulness levels among healthy controls are quite low, preventing an observable saliency effect. The exclusive observation of this result pattern in the Retention outcome could be attributed to the distinct hypothesized memory processes associated with the RAVLT's outcome variables, i.e., acquisition, storage, and retrieval23. In other words, the suggested salience effects may come into play primarily concerning delayed recall processes and not in the acquisition process, which relies more on working memory.

Overall, our findings suggest a comparable discriminant validity of the VR-RAVLT when compared to the GS-RAVLT in distinguishing between PwPD and healthy controls. It is worth noting that these results are particularly interesting since there were no significant differences in MoCA scores between the groups. One may argue that the RAVLT test has superior ability to distinguish between a healthy cohort and a clinical cohort when compared to the MoCA. This assertion is consistent with the findings of Pan et al.24, where the RAVLT was employed as part of a screening battery between healthy controls and people who suffer from Mild Cognitive Impairment (but see Li et al.25 for contradicting results). One possible reason why the RAVLT appears to be more effective in distinguishing between PwPD and healthy individuals is that the RAVLT is a dedicated memory test, while the MoCA covers multiple cognitive domains. This difference may be particularly relevant in the context of PwPD without dementia, as the participants in the present study, where subtle cognitive changes may have only begun to emerge. As the course of PD cognitive decline is yet to be unraveled26,27, further research exploring this potential distinction between the RAVLT and the MoCA among PwPD could prove promising.

The test–retest reliability results reveal consistently high acquisition reliability for both groups in both tests. Notably, the PwPD group exhibits greater retention reliability in the VR-RAVLT, contrasting with the healthy control group where the opposite pattern is observed. Similar to the explanation for within-format retention results, we can apply the same logic here. The heightened reliability of the VR-RAVLT for the PwPD group suggests that the saliency of the VR experience may have conferred a contextual advantage, thereby enhancing the reliability of this test.

The gender disparity observed between the PwPD group and the control group may be attributed to the fact that the risk of developing PD is higher in men compared to women28. This difference might be a confounding factor. Yet, the control analysis reported in the results section excluded this option. It is important to note that PwPD typically exhibit lower performance on the GS-RAVLT compared to healthy controls7,18,29,30. This trend is consistent with both versions of the RAVLT in the current study.

Comparison to the literature

With regards to the GS-RAVLT, the current results align with the existing literature as they demonstrate a reduced performance of PwPD on the GS-RAVLT compared to healthy controls [e.g.,7,18,29,30. For instance, Lyros et al.30 reported higher Acquisition scores among healthy controls compared to PwPD. However, in terms of actual scores, the present GS-RAVLT Acquisition and Retention scores for PwPD (43.9 and 3.3 respectively) not always align with those reported in the literature19,29,31–33, while occasionally showing similar outcomes34,35. For example, Fornari et al.31 reported much lower levels of Acquisition scores (i.e., score = 32.1) among similar H&Y scored PwPD, whereas Alegret et al.34 reported similar scores to the ones reported here (i.e., score = 41.2). Several factors could account for these variations in scores, including differences in inclusion/exclusion criteria and various methodological approaches. This situation emphasizes the importance of establishing standardized methods and normative data for both healthy and clinical cohorts when introducing a new assessment tool like the VR-RAVLT.

While attempting to compare the present discriminant validity results of the VR-RAVLT (or the GS-RAVLT) between healthy individuals and PwPD using ROC curves, we found no existing studies that employed a similar methodology. However, the current findings align with our prior study results, which utilized the VR-RAVLT to discriminate between age groups12, although the current ROC values appear to be lower than our previously reported ROC curves for discriminating between old and non-old groups.

Concerning serial position effects, our present findings align with prior literature, indicating comparable primacy and recency patterns as seen in the current study among PwPD7,36. These results, along these previous reports, imply that primacy and recnecy effect's underlying cognitive mechanisms remain relatively unaffected in non-demented PwPD and indicate of their robustness.

Finally, we did not find studies assessing the test–retest reliability of the RAVLT among PwPD. However, comparing our current results to our previous findings in healthy participants, the reliability among PwPD is somewhat higher for both test formats and outcome measures. This could be explained by the relative homogeneity of the PwPD group, leading to a restriction of range that could inflate reliability measures37.

Recently, Gallagher et al.38 reported on the virtual administration of neuropsychological testing in Parkinson's disease, including the GS-RAVLT, conducted through video conferencing by a human clinician. The results demonstrated relatively good reliability compared to in-person administration. The question then arises: what is the advantage of creating a true VR version of the GS-RAVLT? We suggest that creating a VR environment adaptation allows to simulate real-world scenarios, addressing a limitation of the original test and enhancing its ecological validity. Secondly, a computerized VR version provides the flexibility to easily create, test, and validate variants of the test, tailored for different cohorts. This inclusivity extends to populations such as children or individuals with hearing or speaking disorders, who currently encounter challenges in undertaking the GS-RAVLT.

Ecological validity of the VR-RAVLT

The present findings offer tentative support for an enhanced ecological validity of the VR-RAVLT in comparison to the GS-RAVLT. As previously discussed, the VR settings may have induced a saliency effect, positively influencing the Retention outcomes among the PwPD participants. If so, our study taps on an important problem of traditional neuropsychological testing, that is, their limited capacity to encompass additional factors that may influence memory performance and VR technology might provide the capability to more effectively capture these nuances.

Limitation and future directions

The present study is limited as it was only conducted with a single clinical cohort of PwPD participants, none of whom were diagnosed with PDD. Further, the patients had relatively mild degree of symptoms (i.e., relatively low H&Y scale), which limits our ability to conclude that the VR-RAVLT is suitable for use in general among PwPD. Future studies examining longitudinally the RAVLT trajectories of the two formats may shed light into whether they differ in their prediction of incipient dementia. In addition, in order to understand how well the test works at different stages of cognitive decline, it is crucial to explore the VR-RAVLT's reliability and applicability in PwPD who exhibit mild cognitive impairment or dementia. Future research should therefore assess these aspects of the VR-RAVLT across different stages of cognitive decline.

One limitation of the potential use of the VR RAVLT is that accessibility for individuals with varying levels of visual acuity may be constrained. However, it is important to note that, similar to the traditional P&P RAVLT, the list of words in the VR version is still presented auditorily, and the repetitions are performed verbally. Therefore, we consider the impact of visual limitations within the VR settings to be comparable to that experienced with a human tester.

Future research on the VR-RAVLT should focus on testing with diverse clinical cohorts to enhance its validity and reliability. As mentioned in the introduction, the traditional P&P RAVLT has demonstrated the ability to differentiate between various clinical cohorts [e.g., distinguishing between dementia due to Alzheimer's disease and Parkinson's disease;6,7. Given that our current study and previous research have shown the VR-RAVLT produces similar effects to its P&P counterpart, we hypothesize that the VR-RAVLT will also effectively differentiate between these cohorts. However, this hypothesis requires validation through future studies. It is worth noting that the current VR translation, although designed to be more ecologically valid, still retains a somewhat direct translation approach. For example, the core task of reciting a list of 15 words is preserved. Future adaptation should strive to replicate a more true-to-life context and environment, for instance, an endeavor to learn new language words, a list of names, or similar real-life scenarios.

While ancestry and genetic information of the participants was not collected in the current study, such data would be beneficial for future researchers to compare our findings with those from different genetic and cultural backgrounds. Future studies should include detailed ancestry information to facilitate more comprehensive analyses and comparisons across diverse populations. Subsequent studies would also benefit from differentiating between genetic and idiopathic PD cases, as well as identifying specific genes such as LRRK2 or SNCA that may be involved in the disease process39.

Finally, in recent years it became apparent that assimilation of new technologies has become part of the routine work of clinicians40. Thus, a consideration that needs to be taken into account in the design of VR based platform, such as the one presented in this study, is the simplicity of training of the platform operator.

Conclusions

This study further validates the VR-RVALT for PwPD, extending its established validity in healthy individuals. The present findings indicate that VR may introduce additional nuances to the original test, capturing subtleties that the P&P version cannot. This hints at a potential advancement in enhancing the test's ecological validity, although it's crucial to note that these results require further validation.

Acknowledgements

We extend our gratitude to Prof. Ruth Djaldetti, Prof. Rivka Inzelberg, and Dr. Moshe Bondi for their valuable assistance in acquiring clinical data regarding the PwPD participants.

Author contributions

A.G.—Formal analysis, Writing—original draft; Writing—review and editing. S.K.N—Data curation, Validation; G.Z.—Supervision, Writing—review and editing; M.S.B—Conceptualization, Supervision, Writing—review and editing; M.P.—Conceptualization; Formal analysis; Methodology; Supervision; Validation; Writing—review and editing.

Funding

This work funded in part by financial support from Dr. Marina Nissim and Ms. Judie Stein.

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare no competing interests.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Lezak MD Neuropsychological Assessment 2012 Oxford University Press
Lezak, M. D. et al. Neuropsychological Assessment (Oxford University Press, 2012).
2. Organization, W. H. Parkinson disease (2022). https://www.who.int/news-room/fact-sheets/detail/parkinson-disease.
3. Aarsland D Cognitive decline in Parkinson disease Nat. Rev. Neurol. 2017 13 4 217 231 10.1038/nrneurol.2017.27 28257128
Aarsland, D. et al. Cognitive decline in Parkinson disease. Nat. Rev. Neurol. 13(4), 217–231 (2017).28257128
4. Rey A Rey auditory verbal learning test (RAVLT). L’Examen Clinique en psychologie 1964 PUF
Rey, A. Rey auditory verbal learning test (RAVLT). L’Examen Clinique en psychologie (PUF, 1964).
5. Vakil E Blachstein H Rey AVLT: Developmental norms for adults and the sensitivity of different memory measures to age Clin. Neuropsychol. 1997 11 4 356 369 10.1080/13854049708400464
Vakil, E. & Blachstein, H. Rey AVLT: Developmental norms for adults and the sensitivity of different memory measures to age. Clin. Neuropsychol. 11(4), 356–369 (1997).
6. Tierney MC Recognition memory and verbal fluency differentiate probable Alzheimer disease from subcortical ischemic vascular dementia Arch. Neurol. 2001 58 10 1654 1659 10.1001/archneur.58.10.1654 11594925
Tierney, M. C. et al. Recognition memory and verbal fluency differentiate probable Alzheimer disease from subcortical ischemic vascular dementia. Arch. Neurol. 58(10), 1654–1659 (2001).11594925
7. Tierney MC Use of the Rey Auditory Verbal Learning Test in differentiating normal aging from Alzheimer's and Parkinson's dementia Psychol. Assess. 1994 6 2 129 10.1037/1040-3590.6.2.129
Tierney, M. C. et al. Use of the Rey Auditory Verbal Learning Test in differentiating normal aging from Alzheimer’s and Parkinson’s dementia. Psychol. Assess. 6(2), 129 (1994).
8. Chaytor N Schmitter-Edgecombe M Burr R Improving the ecological validity of executive functioning assessment Arch. Clin. Neuropsychol. 2006 21 3 217 227 10.1016/j.acn.2005.12.002 16554143
Chaytor, N., Schmitter-Edgecombe, M. & Burr, R. Improving the ecological validity of executive functioning assessment. Arch. Clin. Neuropsychol. 21(3), 217–227 (2006).16554143
9. Sunderland A Harris JE Baddeley AD Do laboratory tests predict everyday memory? A neuropsychological study J. Verbal Learn. Verbal Behav. 1983 22 3 341 357 10.1016/S0022-5371(83)90229-3
Sunderland, A., Harris, J. E. & Baddeley, A. D. Do laboratory tests predict everyday memory? A neuropsychological study. J. Verbal Learn. Verbal Behav. 22(3), 341–357 (1983).
10. Lawrence BJ Gasson N Loftus AM Prevalence and subtypes of mild cognitive impairment in Parkinson’s disease Sci. Rep. 2016 6 1 33929 10.1038/srep33929 27650569
Lawrence, B. J., Gasson, N. & Loftus, A. M. Prevalence and subtypes of mild cognitive impairment in Parkinson’s disease. Sci. Rep. 6(1), 33929 (2016).27650569
11. Goldman JG Cognitive impairment in Parkinson’s disease: A report from a multidisciplinary symposium on unmet needs and future directions to maintain cognitive health npj Parkinson's Disease 2018 4 1 19 10.1038/s41531-018-0055-3 29951580
Goldman, J. G. et al. Cognitive impairment in Parkinson’s disease: A report from a multidisciplinary symposium on unmet needs and future directions to maintain cognitive health. npj Parkinson’s Disease 4(1), 19 (2018).29951580
12. Gottlieb A Development and validation of virtual reality-based Rey Auditory Verbal Learning Test Front. Aging Neurosci. 2022 14 980093 10.3389/fnagi.2022.980093 36185470
Gottlieb, A. et al. Development and validation of virtual reality-based Rey Auditory Verbal Learning Test. Front. Aging Neurosci. 14, 980093 (2022).36185470
13. Montreal Cognitive Assessment (MoCA) clinical data. [cited 2024; Available from: https://mocacognition.com/moca-clinic-data/].
14. Goetz CG Movement Disorder Society Task Force report on the Hoehn and Yahr staging scale: Status and recommendations the Movement Disorder Society Task Force on rating scales for Parkinson's disease Mov. Disord. 2004 19 9 1020 1028 10.1002/mds.20213 15372591
Goetz, C. G. et al. Movement Disorder Society Task Force report on the Hoehn and Yahr staging scale: Status and recommendations the Movement Disorder Society Task Force on rating scales for Parkinson’s disease. Mov. Disord. 19(9), 1020–1028 (2004).15372591
15. Broeders M Cognitive change in newly-diagnosed patients with Parkinson's disease: A 5-year follow-up study J. Int. Neuropsychol. Soc. 2013 19 6 695 708 10.1017/S1355617713000295 23544964
Broeders, M. et al. Cognitive change in newly-diagnosed patients with Parkinson’s disease: A 5-year follow-up study. J. Int. Neuropsychol. Soc. 19(6), 695–708 (2013).23544964
16. Erdfelder E Faul F Buchner A GPOWER: A general power analysis program Behav. Res. Methods Instrum. Comput. 1996 28 1 1 11 10.3758/BF03203630
Erdfelder, E., Faul, F. & Buchner, A. GPOWER: A general power analysis program. Behav. Res. Methods Instrum. Comput. 28(1), 1–11 (1996).
17. Hunter H Longitudinal falls data in Parkinson’s disease: Feasibility of fall diaries and effect of attrition Disabil. Rehabil. 2018 40 19 2236 2241 10.1080/09638288.2017.1329357 28573883
Hunter, H. et al. Longitudinal falls data in Parkinson’s disease: Feasibility of fall diaries and effect of attrition. Disabil. Rehabil. 40(19), 2236–2241 (2018).28573883
18. Ibarretxe-Bilbao N Dysfunctions of cerebral networks precede recognition memory deficits in early Parkinson's disease Neuroimage 2011 57 2 589 597 10.1016/j.neuroimage.2011.04.049 21554963
Ibarretxe-Bilbao, N. et al. Dysfunctions of cerebral networks precede recognition memory deficits in early Parkinson’s disease. Neuroimage 57(2), 589–597 (2011).21554963
19. Pellecchia M Insulin-like growth factor-1 predicts cognitive functions at 2-year follow-up in early, drug-naive Parkinson's disease Eur. J. Neurol. 2014 21 5 802 807 10.1111/ene.12137 23551560
Pellecchia, M. et al. Insulin-like growth factor-1 predicts cognitive functions at 2-year follow-up in early, drug-naive Parkinson’s disease. Eur. J. Neurol. 21(5), 802–807 (2014).23551560
20. Koo TK Li MY A guideline of selecting and reporting intraclass correlation coefficients for reliability research J. Chiropr. Med. 2016 15 2 155 163 10.1016/j.jcm.2016.02.012 27330520
Koo, T. K. & Li, M. Y. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J. Chiropr. Med. 15(2), 155–163 (2016).27330520
21. Pedale T Developmental differences in the impact of perceptual salience on short-term memory performance and meta-memory skills Sci. Rep. 2022 12 1 1 13 10.1038/s41598-022-11624-8 34992227
Pedale, T. et al. Developmental differences in the impact of perceptual salience on short-term memory performance and meta-memory skills. Sci. Rep. 12(1), 1–13 (2022).34992227
22. Pedale T Santangelo V Perceptual salience affects the contents of working memory during free-recollection of objects from natural scenes Front. Hum. Neurosci. 2015 9 60 10.3389/fnhum.2015.00060 25741266
Pedale, T. & Santangelo, V. Perceptual salience affects the contents of working memory during free-recollection of objects from natural scenes. Front. Hum. Neurosci. 9, 60 (2015).25741266
23. Vakil E Blachstein H Rey auditory-verbal learning test: Structure analysis J. Clin. Psychol. 1993 49 6 883 890 10.1002/1097-4679(199311)49:6<883::AID-JCLP2270490616>3.0.CO;2-6 8300877
Vakil, E. & Blachstein, H. Rey auditory-verbal learning test: Structure analysis. J. Clin. Psychol. 49(6), 883–890 (1993).8300877
24. Pan F-F A comparative study on the validations of three cognitive screening tests in identifying subtle cognitive decline BMC Neurol. 2020 20 1 8 10.1186/s12883-020-01657-9 31900128
Pan, F.-F. et al. A comparative study on the validations of three cognitive screening tests in identifying subtle cognitive decline. BMC Neurol. 20, 1–8 (2020).31900128
25. Li X The role of the Montreal Cognitive Assessment (MoCA) and its memory tasks for detecting mild cognitive impairment Neurol. Sci. 2018 39 6 1029 1034 10.1007/s10072-018-3319-0 29550982
Li, X. et al. The role of the Montreal Cognitive Assessment (MoCA) and its memory tasks for detecting mild cognitive impairment. Neurol. Sci. 39(6), 1029–1034 (2018).29550982
26. Roheger M Kalbe E Liepelt-Scarfone I Progression of cognitive decline in Parkinson’s disease J. Parkinson's Dis. 2018 8 2 183 193 10.3233/JPD-181306 29914040
Roheger, M., Kalbe, E. & Liepelt-Scarfone, I. Progression of cognitive decline in Parkinson’s disease. J. Parkinson’s Dis. 8(2), 183–193 (2018).29914040
27. Biundo R Weis L Antonini A Cognitive decline in Parkinson’s disease: The complex picture npj Parkinson's Disease 2016 2 1 1 7 10.1038/npjparkd.2016.18
Biundo, R., Weis, L. & Antonini, A. Cognitive decline in Parkinson’s disease: The complex picture. npj Parkinson’s Disease 2(1), 1–7 (2016).
28. Cerri S Mus L Blandini F Parkinson’s disease in women and men: What’s the difference? J. Parkinson's Dis. 2019 9 3 501 515 10.3233/JPD-191683 31282427
Cerri, S., Mus, L. & Blandini, F. Parkinson’s disease in women and men: What’s the difference?. J. Parkinson’s Dis. 9(3), 501–515 (2019).31282427
29. Ramírez-Ruiz B Neuropsychological deficits in Parkinson's disease patients with visual hallucinations Mov. Disord. Off. J. Mov. Disord. Soc. 2006 21 9 1483 1487 10.1002/mds.20965
Ramírez-Ruiz, B. et al. Neuropsychological deficits in Parkinson’s disease patients with visual hallucinations. Mov. Disord. Off. J. Mov. Disord. Soc. 21(9), 1483–1487 (2006).
30. Lyros E Messinis L Papathanasopoulos P Does motor subtype influence neurocognitive performance in Parkinson’s disease without dementia? Eur. J. Neurol. 2008 15 3 262 267 10.1111/j.1468-1331.2007.02046.x 18190508
Lyros, E., Messinis, L. & Papathanasopoulos, P. Does motor subtype influence neurocognitive performance in Parkinson’s disease without dementia?. Eur. J. Neurol. 15(3), 262–267 (2008).18190508
31. Fornari LHT Striatal dopamine correlates to memory and attention in Parkinson’s disease Am. J. Nucl. Med. Mol. Imaging 2021 11 1 10 33688451
Fornari, L. H. T. et al. Striatal dopamine correlates to memory and attention in Parkinson’s disease. Am. J. Nucl. Med. Mol. Imaging 11(1), 10 (2021).33688451
32. Ibarretxe-Bilbao N Hippocampal head atrophy predominance in Parkinson’s disease with hallucinations and with dementia J. Neurol. 2008 255 1324 1331 10.1007/s00415-008-0885-8 18821043
Ibarretxe-Bilbao, N. et al. Hippocampal head atrophy predominance in Parkinson’s disease with hallucinations and with dementia. J. Neurol. 255, 1324–1331 (2008).18821043
33. Junqué C Amygdalar and hippocampal MRI volumetric reductions in Parkinson's disease with dementia Mov. Disord. Off. J. Mov. Disord. Soc. 2005 20 5 540 544 10.1002/mds.20371
Junqué, C. et al. Amygdalar and hippocampal MRI volumetric reductions in Parkinson’s disease with dementia. Mov. Disord. Off. J. Mov. Disord. Soc. 20(5), 540–544 (2005).
34. Alegret M Effects of bilateral subthalamic stimulation on cognitive function in Parkinson disease Arch. Neurol. 2001 58 8 1223 1227 10.1001/archneur.58.8.1223 11493162
Alegret, M. et al. Effects of bilateral subthalamic stimulation on cognitive function in Parkinson disease. Arch. Neurol. 58(8), 1223–1227 (2001).11493162
35. Jozwiak N REM sleep behavior disorder and cognitive impairment in Parkinson’s disease Sleep 2017 40 8 zsx101 10.1093/sleep/zsx101 28645156
Jozwiak, N. et al. REM sleep behavior disorder and cognitive impairment in Parkinson’s disease. Sleep 40(8), zsx101 (2017).28645156
36. Taylor AE Saint-Cyr J Lang A Frontal lobe dysfunction in Parkinson's disease: The cortical focus of neostriatal outflow Brain 1986 109 5 845 883 10.1093/brain/109.5.845 3779372
Taylor, A. E., Saint-Cyr, J. & Lang, A. Frontal lobe dysfunction in Parkinson’s disease: The cortical focus of neostriatal outflow. Brain 109(5), 845–883 (1986).3779372
37. Fife DA Mendoza JL Terry R The assessment of reliability under range restriction: A comparison of α, ω, and test–retest reliability for dichotomous data Educ. Psychol. Meas. 2012 72 5 862 888 10.1177/0013164411430225
Fife, D. A., Mendoza, J. L. & Terry, R. The assessment of reliability under range restriction: A comparison of α, ω, and test–retest reliability for dichotomous data. Educ. Psychol. Meas. 72(5), 862–888 (2012).
38. Gallagher J Validating virtual administration of neuropsychological testing in Parkinson disease: A pilot study Sci. Rep. 2023 13 1 16243 10.1038/s41598-023-42934-0 37758767
Gallagher, J. et al. Validating virtual administration of neuropsychological testing in Parkinson disease: A pilot study. Sci. Rep. 13(1), 16243 (2023).37758767
39. Konno T Siuda J Wszolek ZK Genetics of Parkinson‘s disease: A review of SNCA and LRRK2 Wiad Lek. 2016 69 3 Pt 1 328 332 27486710
Konno, T., Siuda, J. & Wszolek, Z. K. Genetics of Parkinson‘s disease: A review of SNCA and LRRK2. Wiad Lek. 69(3 Pt 1), 328–332 (2016).27486710
40. Senbekov M The recent progress and applications of digital technologies in healthcare: A review Int. J. Telemed. Appl. 2020 2020 1 8830200 33343657
Senbekov, M. et al. The recent progress and applications of digital technologies in healthcare: A review. Int. J. Telemed. Appl. 2020(1), 8830200 (2020).33343657
