
==== Front
Clin Med (Lond)
Clin Med (Lond)
Clinical Medicine
1470-2118
1473-4893
Royal College of Physicians

S1470-2118(24)05426-5
10.1016/j.clinme.2024.100241
100241
Research Article
Utilising accessible and reproducible neurological assessments in clinical studies: Insights from use of the Neurological Impairment Scale in the multi-centre COVID-CNS study
Alam Ali M. a1
Webb Glynn W. a1
Collie Ceryce a
Mariathasan Sashini b
Huang Yun a
Hilton Orla a
Shil Rajish a
Dodd Katherine C. c
Lilleker James B. c
Smith Craig J. cd
Easton Ava ae
Tamborska Arina a
Thomas Rhys H. f
Davies Nicholas W.S. g
Jenkins Thomas M. hi
Zandi Michael b
Benjamin Laura b
Ellul Mark A. a
Solomon Tom aj
Pollak Thomas A. k
Nicholson Tim l
Breen Gerome m
van Wamelen Daniel J. n
Wood Nicholas W. o
Michael Benedict D. benmic@liverpool.ac.uk
a⁎
(on Behalf of the COVID-CNS Group)
a
a Infection Neuroscience Laboratory, Institute of Infection, Veterinary and Ecological Science, University of Liverpool, NIHR HPRU for Emerging and Zoonotic Infection, Liverpool, UK
b National Hospital for Neurology and Neurosurgery, London, UK
c Manchester Centre for Clinical Neurosciences, Northern Care Alliance NHS Foundation Trust, Manchester Academic Health Science Centre, Salford, UK
d Divsion of Cardiovascular Sciences, University of Manchester, Manchester, UK
e Encephalitis International, Malton, UK
f Translational and Clinical Research Institute, Newcastle University, Newcastle upon Tyne, UK
g Department of Neurology, Charing Cross Hospital, London, UK
h Sheffield Institute for Translational Neuroscience, University of Sheffield, UK
i Curtin University, Perth, Western Australia
j The Pandemic Institute, Liverpool, UK
k Department of Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
l Section of Cognitive Neuropsychiatry, Department of Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
m Institute of Psychiatry, Psychology & Neuroscience; Social, Genetic & Developmental Psychiatry Centre, King's College London, London, UK
n Institute of Psychiatry, Psychology & Neuroscience, Department of Neuroimaging, King's College London, London, UK
o Department of Clinical and Movement Neurosciences, University College London, UCL Queen Square Institute of Neurology, London, UK
⁎ Corresponding author. benmic@liverpool.ac.uk
1 Joint first author

06 9 2024
9 2024
06 9 2024
24 5 10024130 7 2024
30 7 2024
27 8 2024
© 2024 Published by Elsevier Ltd on behalf of Royal College of Physicians.
2024

https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Reproducible and standardised neurological assessment scales are important in quantifying research outcomes. These scales are often performed by non-neurologists and/or non-clinicians and must be robust, quantifiable, reproducible and comparable to a neurologist's assessment. COVID-CNS is a multi-centre study which utilised the Neurological Impairment Scale (NIS) as a core assessment tool in studying neurological outcomes following COVID-19 infection. We investigated the strengths and weaknesses of the NIS when used by non-neurology clinicians and non-clinicians, and compared performance to a structured neurological examination performed by a neurology clinician. Through our findings, we provide practical advice on how non-clinicians can be readily trained in conducting reproducible and standardised neurological assessments in a multi-centre study, as well as illustrating potential pitfalls of these tools.

Keywords

COVID-19
Clinical study
Neurological assessment
Neurology
Outcome tool
==== Body
pmcIntroduction

Accurate and reproducible neurological examination is vital in evaluating and classifying neurological conditions in clinical practice. In addition, these clinical assessments are essential tools in neurological research, and are often applied as primary outcome measures of disease.1, 2, 3, 4 When assessing outcomes in neurological studies, it is important to utilise a standardised outcome measurement as it can help with comparing disease processes, eg the neurological sequelae following COVID-19 against the sequalae following meningitis. It also facilitates the combining of studies into meta-analyses, which can feed into data-driven interventions, as well as ensure quantifiable and reproducible results.5,6 Examples such as the National Institutes of Health stroke scale (NIHSS) and the Glasgow coma scale (GCS) demonstrate the utility of these tools to characterise and quantify focused clinical neurological findings in a reproducible form, although their use is limited to specific neurological presentations.3,7

Ensuring effective use of an appropriate neurological assessment tool is particularly important in the context of large multi-centre research studies, given that readily reproducible assessments are required for meaningful comparisons across different sites, settings and populations; this is especially pertinent in the context of heterogenous disease, an exemplar of which is the COVID-Clinical Neuroscience Study (COVID-CNS), which studied the neurological complications of SARS-CoV-2 infection.3 In such settings, neurological assessments are often necessarily performed by non-clinical research staff to enable large cohorts of patients to be assessed across multiple sites within time constraints.

COVID-CNS is a UK multi-centre observational case–control study that assessed the neurological, psychiatric and cognitive complications of COVID-19.8 An important aspect of the study was assessment of neurological examination findings associated with the development of COVID complications. To achieve this at a national scale, a neurological assessment approach was developed whereby all patients underwent a face-to-face clinical assessment involving two neurological assessments. One assessment was a ‘core’ neurological assessment, which could be completed by any appropriately trained research team member, including nurses and research assistants. This was assessed in the same patient relative to a structured ‘clinical’ neurological examination, completed by a clinician, such as a neurology clinical fellow, registrar, or consultant.

The Neurological Impairment Scale (NIS) was chosen as the tool for the core assessment in COVID-CNS. The NIS assesses a combination of neurological impairments and has been applied across a broad range of disabling conditions.4 The NIS is intended to be used both to describe patients on a first assessment and contribute to evaluating recovery (an example of the NIS score sheet and main scoring principles is given in Appendix 1). Good inter-rater agreement has been demonstrated between physicians and multidisciplinary team members, with ratings highly comparable for most items.4 However, the utility of the NIS relative to a structured neurological examination performed by a clinician has not yet been reported.

Aims and objectives

Here, we review the neurological outcomes reported in COVID-CNS in both the ‘core’ examination (which utilised the NIS) in comparison to the ‘clinical’ examination. We aimed to explore two main questions:• How comparable are findings from the NIS to a structured neurological exam when performed by the same clinician?

• How comparable are findings from the NIS when a conducted by a non-clinician to a structure neurological exam performed by a clinician?

By investigating the strengths and weaknesses of the NIS compared to a structured neurological examination, we hope to provide practical insights into utilising standardised neurological assessments to inform future multi-centre large-scale studies.

Methods

Study design

Participants were recruited into the COVID-Clinical Neuroscience Study (COVID-CNS) between October 2020 and October 2022 in accordance with the ethically-approved NIHR Bioresource (East of England–Cambridge Central Research Ethics Committee (Ref 17/EE/0025; 22/EE/0230). All participants gave written informed consent, and all procedures were performed in accordance with the declaration of Helsinki. The purpose of the study was to investigate patients who had been hospitalised with COVID-19 with or without neurological complications. These were defined by the following criteria: neurological disease onset within 6 weeks of acute SARS-CoV-2 infection and no evidence of other commonly associated causes.8

Participants in COVID-CNS had a face-to-face assessment once they recovered from their COVID-19 infection. During this assessment they underwent basic neurological assessment (outlined below), which was divided into two sections (Table 1):1. Core Neurological Assessment – this was completed by appropriately trained research team members who were not clinicians, such as COVID-CNS research assistants / research nurse support and involved completing the NIS.

2. Clinical Neurological Assessment – This was completed by a clinician, who was either a neurology consultant, registrar or a clinical fellow, and comprised completing a full structured neurology examination.

Table 1 Overview of core and clinical neurological assessments used in COVID-CNS.

Table 1	Core neurological assessment	Clinical neurological assessment	
Individual conducting assessment	Group A: Clinician – a neurology consultant, registrar, or clinical fellow
Group B: Non-clinician – a research nurse or a research assistant	Group A and B: a neurology consultant, registrar, or clinical fellow	
Assessment overview	Assessed using the Neurological Impairment Scale containing the following sub-domains:
Speech and language, smell, taste, seeing/vision and hearing
Motor – including gait, upper limb and lower limb power, including ataxia and coordination
Tone/joint range – both upper and lower limb
Sensation – including light touch and proprioception
Cognitive function – including consciousness, orientation, memory and attention	Assessed using a structured neurological examination containing assessment of:
Cranial nerve exam – including vision, smell, taste, hearing and speech
Power – including gait, upper limb and lower limb power
Tone – both upper and lower limb
Sensation – including light touch and proprioception
Coordination – including both truncal and limb ataxia
Basic cognitive testing – involving a modified Abbreviated Mental Test	
Assessment outcomes utilised	Gathered the scores of the 17 items on the NIS, each rated either 0 (none), 1 (mild), 2 (moderate) or 3 (severe), and classified any score above 0 as a positive finding.	Any abnormal findings on neurological examination were classified as a positive finding.	
Training prior to study	In-person and online accessible training video to both clinicians and non-clinicians	In-person and online accessible training video to clinicians	

Participants underwent neurological assessment by one of two ways, depending on the recruiting site (Fig. 1):• Group 1 – Assessment performed by the same clinician for both the core and clinical assessment (in no specified order);

• Group 2 – Core assessment performed by a non-clinician and the clinical assessment performed by a neurology clinician. Both assessors for Group 2 were blinded to the findings of the other.

Fig. 1 Study design showing how core and clinical assessments were undertaken.

Fig. 1

Inclusion criteria

Patients for our study were recruited from 12 UK hospitals in the COVID-CNS study. Exclusion criteria were age <16 years old or if their findings were not thought to be linked to the COVID-19 infection, eg they had pre-existing diagnosis of chronic neurological disease, such as dementia, stroke or multiple sclerosis.

Training for core and clinical assessments

Two open access neurological examination training videos were created prior to assessment (Fig. 2). COVID-CNS staff who were undertaking assessments were trained in the core and clinical neurological assessments with both in-person and online sessions. The core neurological assessment training involved a walk-through of how to conduct the NIS (seen here: https://youtu.be/MJfd4_FtOLA?si=T0QMd0ttR_CXFev), while the clinical assessment training (seen here: https://youtu.be/-3bFWRnAJ58) included a walk-through of the structured neurological exam.Fig. 2 A screenshot of the educational video on conducting the COVID-CNS clinical assessment.

Fig. 2

In these sessions, the principal investigators highlight pertinent signs and symptoms, and specify interpretation of severity scales (eg the difference between moderate and severe motor impairment during an assessment). Video examinations enabled flexible revisiting of teaching materials and direct visualisation of positive findings, previously shown to improve assessment skills.9, 10, 11

Statistical analysis

To assess the utility of using the NIS, we collected outcome data from both the core and clinical neurological assessments. For the core assessment, we gathered the scores of the 17 items on the NIS, each rated either 0 (none), 1 (mild), 2 (moderate) or 3 (severe), giving a total score range 0–50 and classified any score above 0 as a positive finding. Clinical assessment data were also collected and any abnormal findings on neurological examination were classified as positive.

The assessment outcomes for both the core and clinical assessment were classified into the following categories: tone, power (upper limb and lower limb), coordination (including ataxia), sensation (including light touch and proprioception), smell, taste, vision, hearing and speech.

Agreement between Group 1 and Group 2 scores were assessed by reporting Kappa (κ) coefficients for each category assessed. A priori interpretation of the κ statistic used standard rating criteria: 0.00–0.20 slight agreement, 0.21–0.40 fair agreement, 0.41–0.60 moderate agreement, 0.61–0.80 substantial agreement, and 0.81–0.99 almost complete agreement.12 All statistical analyses were conducted using R statistics (R version 4.3.2, epi.kappa package).

Results

Overall, 240 patients were included; 118 in Group 1 and 122 in Group 2. Descriptive data for the groups are shown in Table 2.Table 2 Demographic data of included patients.

Table 2	Group 1 n = 118	Group 2 n = 122	Total n = 240	
Mean age in years
(+/- SD)	56.1 (+/-14.6)	53.6 (+/-15.2)	54.9 (+/-14.9)	
Sex (%)	Female 45 (38.5%)
Male 66 (56.4%)
Unknown 6 (5.1%)	Female 44 (35.7%)
Male 65 (52.8%)
Unknown 14 (11.4%)	Female 89 (37.1%)
Male 131 (54.6%)
Unknown 20 (8.3%)	
Ethnicity	Asian 4 (3.4%)
Black 30 (25.6%)
White 55 (47.0%)
Mixed 3 (2.7%)
Other 17 (14.5%)
Unknown 8 (6.8%)	Asian 7 (5.7%)
Black 5 (4.1%)
White 84 (68.3%)
Mixed 2 (1.6%)
Other 10 (8.1%)
Unknown 15 (12.2%)	Asian 11 (4.6%)
Black 35 (14.6%)
White 139 (57.9%)
Mixed 5 (2.1%)
Other 27 (11.3%)
Unknown 23 (9.6%)	
English as first language (%)	Yes 91 (77.8%)
No 16 (13.7%)
Unknown 10 (8.6%)	Yes 87 (70.7%)
No 18 (14.6%)
Unknown 18 (14.6%)	Yes 178 (74.2%)
No 34 (14.2%)
Unknown 28 (11.7%)	

Group 1 findings

When assessing congruence between positive findings in the core assessment and the clinical assessments within Group 1 (ie the same clinician doing both assessments), we found that the categories with the highest κ co-efficient agreement included power (κ=0.81) (specifically κ=0.76 in the upper limb and κ=0.82 in the lower limb), sensation (κ=0.77), smell (κ=0.87) and taste (κ=0.87). The poorest agreement between the core and clinical assessments were found in tone (κ=0.09) and ataxia (κ=0.28) (Fig. 3).Fig. 3 Kappa co-efficient values in the different neurological assessment categories in Group 1 and Group 2.

Fig. 3

Group 2 findings

When assessing agreement between a non-clinician conducting the core assessment and a clinician conducting a clinical assessment, we found that assessment of power in the upper limb had significant agreement (κ=0.69), as well as smell (κ=0.76). However, only slight agreement was found in assessment of ataxia (κ=0.08). Overall, lower agreement was found between non-clinicians and clinicians (Fig. 3).

Discussion

To ensure robustness as neurological outcome measures, assessments utilised in research must be objective, quantifiable, reproducible, and comparable to a neurologist's assessment.6

How comparable are findings from the NIS to a structured neurological exam when performed by the same clinician?

In COVID-CNS, we found that the NIS had moderate to substantial agreement on many domains of neurological assessment when compared to a structured clinical examination when both were conducted by the same clinician. Specifically, power in both upper and lower limb, smell and taste had almost perfect agreement. Interestingly, elicited neurological signs, such as power, have been reported to have poor inter-rater reliability between neurologists;13 however, in our study, loss of power seems to have good agreement between the NIS and clinical examination. The poorest agreement between the core and clinical assessments were found in tone and ataxia. Verbal feedback from neurologists suggested that the NIS may not necessarily be straightforward to use for those who normally rely on a clinical neurological exam. Specifically, a clinical neurological exam assesses domains such as tone and ataxia through a hands-on, qualitative evaluation (which is often reliant on experience), whereas an assessment using the NIS involves a structured approach which primarily aims to provide a reproducible quantitative output (in the case of tone, the NIS produces a score of 0–3). Therefore, subtle neurological ‘soft’ signs may be missed in the NIS. This could reflect the low agreement in these domains and suggests a weakness of using a standardised tool in assessing neurology. Similar findings have also been found in studies assessing modified versions of the NIHSS.14,15 Often ataxia can be subtle on examination, and may involve the limbs, trunk or cranial innervation. In the NIS, coordination and ataxia are classified under the motor control section (where hemiparesis and weakness would be assessed), which focuses on the limbs primarily. Therefore, subtle truncal or cranial signs may potentially be missed in the NIS as these are not specifically mentioned in other sections, and then be subsequently picked up in a structured clinical examination. Overall, however, at least fair agreement was found in all categories (Fig. 3), suggesting that the NIS maybe be a useful standardised neurological assessment tool when utilised by clinicians. In practice, using the NIS alongside a neurological exam can complement qualitative clinical assessment with objective and reproducible data that may be tracked over time.

How comparable are findings from the NIS when a conducted by a non-clinician to a structure neurological exam performed by a clinician?

We found that non-clinicians generally demonstrated at least fair agreement across many domains when comparing their NIS assessment to clinicians’ clinical assessments. Certain aspects of the neurological assessments did, however, show only slight agreement, and overall there was poorer agreement compared to the assessments being done by the same clinician. This emphasises the importance of interpreting assessment tools in the context of the neurological syndrome under evaluation. For example, elicited signs such as lower limb power, tone, ataxia, sensation and proprioception only had slight to moderate agreement. Power may be poorly assessed by scales such as the NIS when conducted by non-clinicians due to the subjective interpretation of weakness, as well as cooperation of patients; whereas findings such as tone are reliant on technique to identify subtle clinical signs such as hypotonia. Of note is the extremely low agreement of ataxia and proprioception. These areas of poor agreement may suggest that the NIS requires modification to help increase identification of signs such as ataxia and limits its use in assessing pathologies with proprioceptive sensory deficits. Proficiency and experience are vital in eliciting these signs and standardised assessment scales may well under-report such neurological findings. Interestingly, lower limb motor assessment had worse agreement when compared to the upper limb, perhaps suggesting that more sensitive positive findings on lower limb examinations were missed such as distal weakness, though other aspects such as cooperation of the patient and technique of examiner may again have contributed. Nevertheless, many categories showed moderate or substantial agreement (Fig. 3), illustrating that there may be utility in non-clinicians using the NIS as a standardised assessment tool in neurological studies.

The importance of reproducible and accessible neurological assessment tools

Given the time constraints on principal or associate principal investigators (eg due to clinical commitments), standardised examination tools, such as the NIS, can streamline the assessment of a large number of patients in a reproducible manner. Furthermore, as we have demonstrated with the NIS, assessment tools can be undertaken by the wider MDT with good reliability for many neurological domains. As more large-scale studies in neurology are conducted, further evaluation in the operator-independent utility of neurological assessment tools is vital in ensuring that the maximum number of patients are assessed in a standardised manner.

Assessment tools can be particularly important in studying neurological conditions in cohorts in low-resource settings, especially given that the WHO estimates in some settings there are 0.3 neurologists per million population.16 Furthermore, by utilising standardised neurological assessment, studies in low-resource settings can be compared to those in high-resource settings to better understand disparities in neurological outcomes. In COVID-CNS, videos were utilised to help in training co-investigators to improve consistency between the multiple centres when reporting outcomes. Video tools can be utilised to improve accessibility of outcomes scales in neurological studies globally. Notably, however, very few assessment tools are available in more than one language, which impedes their utility in research beyond English-speaking populations.17

Finally, by evaluating the utility of neurological tools such as the NIS, we can identify strengths and weaknesses in these assessments. For example, the poor agreement in the assessment of lower limb power, coordination, ataxia and sensation in the NIS can have important effects on population-based studies of conditions such as neuropathies and neuronopathies,18,19 where findings such as sensory neuropathies or ataxia may be under-reported. In such cases, findings obtained using these tools must be evaluated in the context of the operator undertaking the neurological assessment.

Limitations

There are several limitations that must be considered in our study. Firstly, not all clinical neurological assessments were completed by consultant neurologists, and therefore there may be incorrect or under-reported findings within the structured neurological examinations. Indeed, previous studies have highlighted the inter-rater variability among consultant neurologists when it comes to clinical examination.13 Furthermore, though training was provided in both the core and clinical examination assessments, these were not exhaustive and better agreement scores may well have been found if substantial and recurring training in the NIS was given to both clinicians and non-clinicians. Our findings focus on the utility of the NIS as a neurological assessment tool in the context of COVID-19, and our findings may not be reproducible in other settings using separate tools studying other neurological pathologies. Finally, in group 1, the clinician may well be biased in their reporting of positive findings in the clinical assessment given they had first completed the core assessment, potentially incorrectly increasing the agreement scores between them.

Conclusion

COVID-CNS utilised the NIS as a tool to assess the neurological morbidity following infection. This involved developing open access video-based walkthroughs on conducting the core and clinical neurologist assessments. Our analysis illustrates that the NIS has good agreement when compared to a clinical neurological examination, and that non-clinicians had mostly above fair agreement when comparing their assessment to clinicians. Certain aspects of the neurological assessments did, however, show only slight agreement, such as lower limb power, coordination and sensation. Neurological assessment sets can provide an accessible and reproducible tool for many parameters which can be used successfully in multi-centre neurological studies, but findings should be interpreted in the context of the neurological syndrome under evaluation.

Summary box

• Effective use of an appropriate neurological assessment tool is important in assessing outcomes in multi-centre research studies, given that readily reproducible assessments are required for meaningful comparisons across different sites, settings, and populations.

• Often these assessment tools are completed by non-clinicians and must be comparable to assessment conducted by neurologists.

• COVID-CNS illustrates how the Neurological Impairment Scale has good agreement when compared to a clinical neurological examination conducted by a neurologist, and that non-clinicians generally demonstrated at least fair agreement when comparing their assessment to clinicians.

• Certain aspects of the neurological assessments did, however, show only slight agreement and this emphasises the importance of interpreting assessment tools in the context of the neurological syndrome under evaluation.

Author contributions

Conceptualisation: Ali M Alam, Glynn Webb, Daniel J van Wamelen, Benedict D Michael; Methodology: Ali M Alam, Glynn Webb, Yun Huang, Mark Ellul, Thomas A Pollack, Daniel J van Wamelen, Benedict D Michael; Formal analysis: Ali M Alam, Glynn Webb, Rajish Shil; Data Curation: Glynn Webb, Ceryce Collie, Daniel J van Wamelen, Benedict D Michael; Writing - Original Draft: Ali M Alam, Glynn Webb, Ceryce Collie, Sashini Mariathasan, Yun Huang, Orla Hilton, Rajish Shil; Writing - Review & Editing: Ali M Alam, Glynn Webb, Ceryce Collie, Sashini Mariathasan, Yun Huang, Orla Hilton, Rajish Shil, Katherine C Dodd, James B. Lilleker, Craig J. Smith, Ava Easton, Arina Tamborska, Rhys H Thomas, Nicholas WS Davies, Thomas M Jenkins, Michael Zandi, Laura Benjamin, Mark A Ellul, Tom Solomon, Thomas A. Pollak, Tim Nicholson, Gerome Breen, Daniel J van Wamelen, Nicholas Wood, Benedict D Michael; Supervision: Gerome Breen, Daniel J van Wamelen, Nicholas Wood, Benedict D Michael.

Declaration of competing interest

T.S. is the Director of The Pandemic Institute which has received funding from Innova and CSL Seqirus and Aviva and DAM Health. T.S. was an advisor to the GSK Ebola Vaccine programme and the Siemens Diagnostic Programme. T.S. Chaired the Siemens Healthineers Clinical Advisory Board. T.S. Co-Chaired the WHO Neuro-COVID task force and sat on the UK Government Advisory Committee on Dangerous Pathogens, and the Medicines and Healthcare Products Regulatory Agency (MHRA) Expert Working Group on Covid-19 vaccines. T.S. Advised to the UK COVID-19 Therapeutics Advisory Panel (UK-TAP). T.S. was a Member of COVID-19 Vaccines Benefit Risk Expert Working Group for the Commission on Human Medicines (CHM) committee of the Medicines and Healthcare products Regulatory Agency (MHRA). T.S. has been a member of the Encephalitis Society since 1998 and President of the Encephalitis Society since 2019.

Appendix Supplementary materials

Image, application 1

Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.clinme.2024.100241.
==== Refs
References

1 Jennett B. Bond M. Assessment of outcome after severe brain damage Lancet 1 7905 1975 480 484 46957
2 Kurtzke J.F. Rating neurologic impairment in multiple sclerosis: an expanded disability status scale (EDSS) Neurology 33 11 1983 1444 1452 6685237
3 Lyden P.D. Lu M. Levine S.R. Brott T.G. Broderick J. NrSS G.roup A modified National Institutes of Health Stroke Scale for use in stroke clinical trials: preliminary reliability and validity Stroke 32 6 2001 1310 1317 11387492
4 Turner-Stokes L. Thu A. Williams H. Casey R. Rose H. Siegert R.J. The Neurological Impairment Scale: reliability and validity as a predictor of functional outcome in neurorehabilitation Disabil Rehabil 36 1 2014 23 31 23721497
5 Williamson P.R. Altman D.G. Bagley H. The COMET Handbook: version 1.0 Trials 18 Suppl 3 2017 280 28681707
6 Hobart J.C. Lamping D.L. Thompson A.J. Evaluating neurological outcome measures: the bare essentials J Neurol Neurosurg Psychiatry 60 2 1996 127 130 8708638
7 Teasdale G. Jennett B. Assessment of coma and impaired consciousness. A practical scale Lancet 2 7872 1974 81 84 4136544
8 Group C.-C.S. COVID-19 Clinical Neuroscience Study (COVID-CNS) 2023
http://www.covidcns.org.

9 Lee H. Choi A. Jang Y. Lee J.I. YouTube as a learning tool for four shoulder tests Prim Health Care Res Dev 20 2018 e70 30375303
10 Regmi K. Jones L. A systematic review of the factors - enablers and barriers - affecting e-learning in health sciences education BMC Med Educ 20 1 2020 91 32228560
11 Shikino K. Rosu C.A. Yokokawa D. Flexible e-learning video approach to improve fundus examination skills for medical students: a mixed-methods study BMC Med Educ 21 1 2021 428 34389012
12 Landis J.R. Koch G.G. The measurement of observer agreement for categorical data Biometrics 33 1 1977 159 174 843571
13 Thaller M. Hughes T. Inter-rater agreement of observable and elicitable neurological signs Clin Med (Lond) 14 3 2014 264 267 24889570
14 Meyer B.C. Lyden P.D. The modified National Institutes of Health Stroke Scale: its time has come Int J Stroke 4 4 2009 267 273 19689755
15 Anderson E.R. Smith B. Ido M. Frankel M. Remote assessment of stroke using the iPhone 4 J Stroke Cerebrovasc Dis 22 4 2013 340 344 22018507
16 Mohammadi D. NeuroAccess: breaking the cycle in resource-poor settings Lancet Neurol 13 12 2014 1170 1171 25453454
17 Ferreira V.R. Metting E. Schauble J. Seddighi H. Beumeler L. Gallo V. eHealth tools to assess the neurological function for research, in absence of the neurologist - a systematic review, part I (software) J Neurol 2023
18 Kazamel M. Stino A.M. Smith A.G. Metabolic syndrome and peripheral neuropathy Muscle Nerve 63 3 2021 285 293 33098165
19 Kirthi V. Perumbalath A. Brown E. Prevalence of peripheral neuropathy in pre-diabetes: a systematic review BMJ Open Diabet Res Care 9 1 2021
