
==== Front
BMC Neurol
BMC Neurol
BMC Neurology
1471-2377
BioMed Central London

3824
10.1186/s12883-024-03824-8
Research
Validity and reliability of the Pain Assessment in Impaired Cognition 15 (PAIC15) observation scale in persons with aphasia
de Vries N. J. n.j.de_vries@lumc.nl

12
Smaling H. J. A. 13
van der Steen J. T. 14
Achterberg W. P. 123
1 https://ror.org/05xvt9f17 grid.10419.3d 0000 0000 8945 2978 Department of Public Health and Primary Care, Leiden University Medical Center, PO Box 9600, Hippocratespad 21, Zone V0-P, Leiden, 2300 RC the Netherlands
2 TOPAZ Geriatric Rehabilitation Center Revitel, Leiden, the Netherlands
3 https://ror.org/05xvt9f17 grid.10419.3d 0000 0000 8945 2978 University Network for the Care sector South Holland, Leiden University Medical Center, Leiden, the Netherlands
4 https://ror.org/05wg1m734 grid.10417.33 0000 0004 0444 9382 Department of Primary and Community Care, and Radboudumc Alzheimer Center, Radboud University Medical Center, Nijmegen, the Netherlands
5 9 2024
5 9 2024
2024
24 31929 3 2024
26 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Background

The use of self-report pain scales in persons with aphasia can be challenging due to communication and cognitive problems, while for assessing pain self-report pain is considered the gold standard (Harrison RA, Field TS. Post stroke pain: identification, assessment, and therapy. Cerebrovasc Dis. 2015;39(3–4):190–201.). An observational scale may be used as an alternative. This study examines the validity and reliability of the observational Pain Assessment in Impaired Cognition (PAIC15) scale in persons with aphasia.

Methods

Persons with aphasia were observed during rest and transfer by two observers using the PAIC15. The PAIC15 comprises 15 items covering the three domains of facial expressions, body movements, and vocalizations. When able, the participant completed four self-report pain scales after each observation. The observations were repeated within one week. For criterion validity, correlations between the PAIC15 and self-report pain scales were calculated and for construct validity, three hypotheses were tested. Reliability was determined by assessing internal consistency, and intra- and interobserver agreement.

Results

PAIC15 observations were obtained for 71 persons (mean age 75.5 years) with aphasia. Fair positive correlations (rest: 0.35–0.50; transfer: 0.38–0.43) were reported between PAIC15 and almost all self-report pain scales. Results show that significantly more pain was observed in persons with aphasia during transfer than during rest. No differences were found for observed pain between persons with aphasia who use pain medication and those without, or persons who have joint diseases compared to those without. Results showed acceptable internal consistency. Intra- and interobserver agreement was high for most PAIC15 items, particularly for the domains body movements and vocalizations during rest and transfer.

Conclusions

Recognition of pain in persons aphasia using the PAIC15 showed mixed yet promising results.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12883-024-03824-8.

Keywords

Aphasia
Self-report pain
Pain scales
Pain observation instrument
ZorgondersteuningsfondsPROM-6 PROM-6 PROM-6 PROM-6 issue-copyright-statement© BioMed Central Ltd., part of Springer Nature 2024
==== Body
pmcIntroduction

Self-report pain scales are commonly used to assess pain in patients with aphasia. Examples are the the Numerical Rating Scale (NRS) [1], the Visual Analog Scale (VAS) [2] and Faces Pain Scale (FPS) [3]. Self-report pain scales require the person to be able to understand verbal and written instructions and to apply this information in his or her response, which limits the use in persons with aphasia. Persons with moderate to severe aphasia are also often excluded from pain research, which makes interpretation of applicability, usefulness and best practices in pain assessment in aphasia difficult, although very relevant [4, 5]. However, stroke patients with mild to moderately-severe aphasia have pain just as often as stroke patients without aphasia (e.g. due to shoulder pain and central pain) [6].

Smith and Bottemiller [7] found that 14% of stroke patients were not able to complete the FPS or NRS. Capacity to complete these scales was associated with the severity of stroke and severity of aphasia. Most studies focused on patients with mild to moderate aphasia [8]. Also, despite varied self-report pain scales, stroke patients are less likely than age-matched controls to be able to complete these pain scales [9, 10]. Evidently, an appropriate alternative method of assessment of the presence of pain in persons with aphasia who are unable to self-report is needed. An alternative to self-report could be the observation of a person’s behavior, as is common in patients with cognitive impairment [11, 12]. Observational pain scales have been used successfully as an alternative to self-report pain scales in people with advanced dementia [13].

In 2011, a European Cooperation in Science and Technology (EU-COST) initiative collaborated to improve pain assessment in persons with impaired cognition. This international multidisciplinary team of experts from 16 countries developed a universal meta-tool for the assessment of pain in persons with cognitive impairment. This meta-tool, the Pain Assessment in Impaired Cognition (PAIC15), is an observational instrument that includes the best items from existing pain scales to observe pain in persons with impaired cognition. The PAIC15 has shown satisfactory psychometric qualities in patients with impaired cognition, mostly with dementia [14, 15]. This pain observation instrument is available in 10 languages and comes with an internet-based E-learning module in three languages (German, Dutch, and English: https://paic15.com/en/e-training-en/). The PAIC15 is therefore a potentially suitable alternative for assessing pain in patients who are unable to self-report, such as those with aphasia [16]. This study aims to answer the following research question: ‘What is the validity and reliability of the Dutch version of PAIC15 in persons with aphasia?’

Methods

Study design

The current study was an observational cohort study to determine the validity and reliability of the Dutch version of PAIC15 in persons with aphasia. Persons with aphasia were observed using PAIC15 during rest and transfer. Rest situations could be lying in bed or sitting in a (wheel)chair. Transfer situations include physical moves from bed to (wheel)chair, repositioning in bed or a short walk. Observations were conducted by two observers and repeated within one week. The data were collected during the COVID19-pandemic between May 2019 and July 2021.

Participants

Speech and language therapists from 19 nursing home organizations in the Netherlands invited the persons with aphasia to participate in the study. The nursing home organizations participated in the University Network for the Care sector – South Holland (UNC-ZH). Further, we used personal networks to invite nursing homes to participate in the study. Inclusion criteria were: residing in a nursing home in a geriatric rehabilitation department or a unit for patients with chronic physical impairments, age 18 years or older, sufficient comprehension of the Dutch or English language before onset of aphasia, and diagnosed with aphasia regardless of cause or severity. A score of ≤ 68 on the ScreeLing [17] or ≥ 7 on the TokenTest [18] implies aphasia. If diagnostic examination was not possible, the speech and language therapist’s clinical judgement was decisive. Persons were excluded if they had a delirium, severe psychiatric disease, dementia, or a life expectancy ≤ 6 months according to the primary responsible physician.

Instruments

Questionnaires 1 and 2

Characteristics of persons with aphasia were assessed with two questionnaires. A informal caregiver or legal representative or the speech and language therapist, if possible together with the person with aphasia, completed the brief questionnaire 1 with questions about persons, hand dominance, and length of stay in the nursing home. Questionnaire 1 is showed in Additional file 1 [see Additional file 1]. The speech and language therapist collected demographic characteristics and reported other more medical characteristics of the aphasia and pain treatment using questionnaire 2. Questionnaire 2 is showed in Additional file 2.

Pain observation scale

Pain symptoms were observed for five to a maximum of ten minutes using the validated Dutch version of the PAIC15 [15, 16]. The PAIC15 includes fifteen items; five in each of the three domains of facial expressions, body movements and vocalizations. Scoring options are ‘not at all’ (0), ‘slight degree’ (1), ‘moderate degree’ (2), ‘great degree’ (3) and ‘not scoreable’ (X). For example, the first item ‘frowning’ is described as moving eyebrows downwards and contracting them. ‘Not at all’ is scored when frowning does not occur during the observation. If this item cannot be assessed (e.g., the person turns their head away), it is rated as: ‘not scoreable’. ‘Slight degree’ (1) is scored when frowning is observed but only briefly or with little intensity; ‘great degree’ (3) when frowning is observed frequently or continuously, ‘moderate degree’ (2) when this item is not constantly observed, but more frequent than briefly. The PAIC15 e-learning provides clear instructions on how to score each item.

Summed total scores range from 0 to 45; 0–15 for each domain. For the statistical analysis, all scores X (not scorable) were regarded as 0 [19, 20]. The observers were students of clinical neuropsychology or medicine, and nurses or speech and language therapists. They received 1.5 h of instruction and completed the PAIC15 e-learning (https://paic15.com/en/e-training-en/).

Self-report pain scales

The four self-report pain scales used were: the vertical NRS, VAS, FPS and a combined scale. The NRS ranges from 0 (no pain) to 10 (worst pain imaginable) (Hjermstad, 2011). The VAS consists of a 10-centimetre line with extremes labelled ‘no pain’ and ‘unbearable pain’. The person is asked to locate the pain intensity on the line. Half a centimetre is rounded up to whole numbers, e.g., 3.5 is counted as 4 centimetres [2]. The FPS comprises six coloured cartoon-faces with expressions no pain (dark green smiling face) to worst pain (dark red sad face) with the values 0, 2, 4, 6, 8, and 10 (Kim, 2006). An additional file shows the sel-report combined pain scale. This combined scale combines the self-report pain scales FPS and NRS [see Additional file 3]. This scale consists of the numbers zero to ten, coloured smiley faces, and written expressions of pain displayed along a vertical line. All self-report pain scales were offered in a vertical form for use in case of visual problems such as neglect or hemianopsia post stroke. The order of the first three scales was randomized, and the final self-report scale was always the combined scale.

Procedure

Questionnaires 1 and 2 were completed on paper before the observations and returned in a closed envelope. Persons with aphasia were observed during rest and transfer twice within 7 days by the same two observers. The observations were performed by trained research assistants who were not familiar with the person with aphasia.

Each observation was performed by two observers (A and B) independently (blinded). The observers were also blind with respect to the questionnaires and self-report pain scales. First, the participant’s language comprehension was checked using the FPS. The participant was asked: ‘Imagine you have no pain now, could you indicate which face on this scale fits this experience?’ and ‘Imagine you have a severe headache at this moment, which face on this scale fits this situation?’. If these questions were answered correctly, language comprehension to complete the self-report pain scales was assumed to be sufficient. If these questions could not be answered and self-report pain scales could not be completed, only PAIC15 was used during the observations. Next, the observers observed the participant for a minimum of 5 and a maximum of 10 min during rest and completed the PAIC15 form independently. Afterwards, if applicable, the participant completed the four self-report pain scales. This procedure was repeated during transfer. The participants were observed during transfer for a minimum of 5 min, even when the transfer sometimes took less time. The procedure was repeated within 7 days by the same observers. After both observations on measurement 1, the observers discussed the independent observations during rest and transfer and jointly completed a new observation form. This new observation form, with the consensus scores of PAIC15, was completed to minimize the risk of behaviour being overlooked and for quality purposes. If the observation during transfer was carried out first, the observation during rest took place after 30 min, to prevent the transfer influencing the observation during rest.

Statistical analysis

Descriptive statistics of the PAIC15

First, we perform general descriptive statistics of the PAIC15 consensus scores and the self-report pain scales if these were used. Because of non-normal distribution, data were expressed as medians with interquartile range (IQR). Second, the presence of responses of the individual PAIC15 items of the consensus scores were examined, and reported in percentages, during rest and transfer. Floor or ceiling effects are defined as ≥ 15% of PAIC15 total scores scored the lowest (0: not at all) or highest possible score (3: great degree) [21]. More than 5% missing scores of items per observation form were discussed, reported, and are not imputed. No PAIC15 observation form and no person was excluded.

Criterion validity

Regarding criterion validity, we expected moderate correlations between PAIC15 and the four self-report pain scales. Because the data was not normally distributed, Spearman’s correlation coefficient and a 95% confidence interval (CI) were used to calculate the correlations between the PAIC15 consensus scores and the four self-report pain scores of measurement 1 in order to determine criterion validity. To describe the strength of the correlation we used: less than 0.30 is poor, 0.3 to 0.5 is fair, 0.6 up to 0.8 moderately strong, and 0.80 and higher is very strong (Chan, 2003). See Table 1 for the definitions of types of validity adapted from COnsensus based Standards for the selection of health Measurement INstruments (COSMIN) as applied in this study [22].

Table 1 Definitions of types of validity

Measurement property + adapted COSMIN definition:	
Criterion validity = the degree to which the score on PAIC15 is an adequate reflection of another well-established self-report pain measure.	
Construct validity/ hypotheses testing = the degree to which the PAIC15 scores are consistent with hypotheses (for instance, relationships to scores of other measures or observer report, or differences between relevant groups) based on the assumption that the PAIC15 validly assesses the construct to be measured.	

Construct validity

To determine construct validity, 3 hypotheses were tested:

More pain is expected during transfer compared to rest in persons with aphasia.

To assess the degree to which the PAIC15 is capable of measuring pain in persons with aphasia, we compared results of the PAIC15 between rest and transfer. Similar research in persons with dementia reported more observed pain during ADL compared to rest [23–25].

2) More pain is expected when persons with aphasia used pain medication compared to persons who did not use pain medication.

Also, research on pain in dementia reports more observed pain in persons who used pain medication compared to persons who do not used pain medication [26, 27] Persons still experience pain, even when they receive pain medication. Additionally, a study of hospitalized persons with dementia (n = 108) who experienced pain (assessed with Pain Assessment in Advanced Dementia (PAINAD)) found that 60% of those persons had received pain medication compared to 40% who did not receive pain medication (Boltz et al., 2021).

3) More pain is expected in persons with aphasia who have joint disease such as osteoarthritis or rheumatism versus those without joint disease.

Osteoarthritis was the most common joint disease, and joint pain was among the most frequent pain syndromes in Europe [28–30]. It is expected that persons with aphasia and joint disease will have more pain than persons without joint disease, due to the risk of increased pain from joint problems and the difficulty in communication due to aphasia.

First, a non-parametric Wilcoxon signed-rank (paired) test was used to examine whether more pain was observed during rest than during ADL. Subsequently, Mann-Whitney U tests were used to investigate if patients with aphasia who use pain medication experienced more pain than those without pain medication, and whether patients with aphasia and joint pain had more pain versus those without joint pain.

Reliability

The reliability of PAIC15 in persons with aphasia was determined by assessing internal consistency, intraobserver and interobserver agreement.

Internal consistency

The internal consistency of the PAIC15 of observers A and B together, during measurement 1, measurement 2, and measurements 1 and 2 together (consensus scores) was examined using Cronbach’s alpha. Cronbach’s α-values ranging from 0.70 to 0.95 are generally considered acceptable (Bland & Altman, 1997).

Intraobserver and interobserver agreement

The intraobserver and interobserver agreement of the individual items of PAIC15 were analysed using percentage agreement [21, 31, 32]. Percentage agreement is more suitable than for example Cohen’s kappa and interpretation by clinicians is more straightforward [32]. Cohen’s kappa is a relative measure of reliability, whereas percentage agreement is an absolute measurement. In clinical practice, the probability that another rater gives the same answers is of interest to healthcare professionals. Therefore, to assess intraobserver agreement, percentage agreement was calculated between the responses of each of the observers on measurements 1 and 2 during rest and transfer. Interobserver agreement was examined using percentage agreement between the PAIC15 4-point scores of observer A on measurements 1 and 2 compared to the scores of observer B on measurements 1 and 2 during rest and transfer. Percentage agreement was also calculated with dichotomized scores (0 = absent; 1,2,3 = present) of the PAIC15 scores of both observers on both measurements during rest and transfer. These percentage agreements of the dichotomized scores were compared with the percentage agreements of the PAIC15 scores using the 4-point scale. Percentage agreement below 70% was regarded as poor and percentage agreement of ≥ 70% was considered high [32]. The analyses were performed using IBM SPSS Statistics version 29 for Windows, 2022.

Results

The study flowchart is presented in Fig. 1 [see Fig. 1]. Data was collected during the COVID-19 pandemic and inclusion of persons with aphasia and collecting data took longer than expected due to the closure of a department of participated nursing homes or quarantine. Speech and language therapists of 14 nursing home organizations invited 95 persons with aphasia to participate; 82 persons with aphasia were included. Pain observations were performed by trained speech and language therapists (N = 4), nurses (N = 8), and trained master’s students (N = 7).Fig. 1 Flowchart inclusion of nursing homes and persons with aphasia. a: SLP = Speech Language Pathologist often called speech and language therapist; b: nursing homes participated if one or two speech language therapist participated; c: Persons with aphasia without a diagnosis of aphasia, psychiatric disorder, or delirium; d: observations took place at a time when no transfer took place (for example because the person was in bed or wheelchair). NRS = Numeric Rating Scale; FPS = Faces Pain Scale; VAS = Visual Analoque Scale

The sample characteristics of the persons with aphasia are shown in Table 2. Almost two-thirds of the persons received pain medication (62%) and were able to complete at least one self-report pain scale (65%). Osteoarthritis or rheumatism were present in 9 (13%) patients (See Table 2).

Table 2 Characteristics of the 71 participating persons with aphasia

	Mean [SD] range, or % (n)	
Age		75.5 [10.6]

40–92

(n = 71)

	
Sex	female

male

	63 (45)

37 (26)

	
Nationality	Dutch

Western migration background

Non-western migration background

Missing

	91 (62)

7 (5)

1 (1)

(3)

	
Level of educationa	Lower

Medium

High

Missing

	49 (33)

21 (14)

30 (20)

(4)

	
Cause of aphasiab	Stroke

Tumor

Trauma

	97 (69)

1 (1)

1 (1)

	
Hand dominance	Right

Left

Missing

	93 (62)

10 (5)

(4)

	
Total duration of hospitalization (months)	(n = 66)	11.8 [24]

3-123

	
Pain medication	Yes

No

Missing

	63 (44)

39 (26)

(1)

	
Joints diseases (osteoarthritis/ rheumatism)	Yes

No

Missing

	14 (9)

86 (55)

(7)

	
Complete self-report pain scales	Yes

No

Missing

	66 (46)

34 (24)

(1)

	
aInternational Standard Classification of Education (ISCED): Lower = 8 years of primary and special primary education; prevocational secondary education; lower secondary vocational training and assistant’s training. Medium = upper secondary education, (basic) vocational training, middle management and specialist education. Higher = higher education, 4-year education at universities of applied sciences and research universities; doctoral degree programs at research universities (UNESCO, 2012)

bThe percentages do not always sum up to 100, due to rounding to whole decimal places

Descriptive statistics of the PAIC15

The descriptive statistics of PAIC15, based on the PAIC15 consensus scores of measurement 1, and the self-report pain scales in persons with aphasia are presented in Table 3. See Additional file 4 for the descriptive statistics of the individual observations using PAIC15 of observers A and B [see Additional file 4]. Table 4 shows the presence of responses on the individual PAIC15 consensus scores in percentages during rest and transfer. During rest, prevalence of all PAIC15 items was low except for the facial expression item ‘opening mouth’, which had a prevalence of ≥ 30%. More items with higher prevalence were found during transfer; three items in the domains facial expressions and one item in the domain vocalizations showed a prevalence of ≥ 30%. All other items had a lower prevalence than 30%, during rest and during transfer (See Table 4). In most cases, the item ‘resisting care’ was ‘not scoreable’ during rest because healthcare professionals were often not present or not providing care. Both during rest and transfer, there is a floor effect with frequencies higher of 15% on score ‘0-not at all’. No ceiling effect emerged. Less than 5% of responses were missing.

Table 3 Descriptive statistics of the PAIC15a consensus scoresb and self-report pain scalesc in persons with aphasia during assessment 1

Instrument	N	Median
(IQR)	Observed Range	
During rest:	71			
PAIC15 total score (range 0-45)		1 (1-3)	0-21	
Self-report pain scales  (range 0-10)	
 FPS	46	2 (0-4)	0-10	
 NRS	46	2 (0-4)	0-10	
 VAS	45	1 (0-4)	0-8	
 Combination scale	46	2 (0-4)	0-10	
During transfer:	
 PAIC15 total score (range 0-45)	70	3 (2-6)	0-18	
Self-report pain scales (range 0-10)	
 FPS	43	2 (0-4)	0-10	
 NRS	43	2 (0-4)	0-9	
 VAS	42	1 (0-4)	0-9	
 Combination scale	43	2 (0-4)	0-9	
 Days between assessment 1 and 2	62	3 (2-5)	1-7	
IQR Interquartile range, FPS Faces pain scale, NRS Numeric rating scale, VAS Visual analogue scale

aPain Assessment in Impaired Cognition with 15 items, subdomain ranges of 0-15 and a total range of 0-45

bThe consensus scores of PAIC15 was based on consensus after discussing scores after independent observations during rest and transfer on day 1

cThe range of self-report pain scales is: 0-10

Table 4 Scores per item of PAIC15a consensus scores (in percentages) during rest (N = 71) and during transfer (N = 70) in patients with aphasia

Score:
Items:	Not scoreable	0
not at all	1
slight degree	2
moderate degree	3
great degree	
Facial expressions	Rest
n = 71	Transfer
n = 70	Rest n = 71	Transfer
n = 70	Rest
n = 71	Transfer
n = 70	Rest
n = 71	Transfer
n = 70	Rest
n = 71	Transfer
n = 70	
1 Frowning			70	49	27	38	3	11			
2 Narrowing eyes			87	70	11	23	1	6			
3 Raising upper lip			93	68	4	28	3	3			
4 Opening mouth			54	31	42	55	3	13	1		
5 Looking tense			69	45	28	41	1	13	1		
Body movements	
 6 Freezing			94	72	4	25	1	1			
 7 Guarding		1	90	87	7	6	1	4	1		
 8 Resisting care	70	43	30	54		3					
 9 Rubbling			93	93	3	3	3	3	1		
 10 Restlessness			89	90	9	7	3	1			
Vocalizations				n = 69		n = 69		n = 69		n = 69	
11 Using pain-related-words			97	80	1	7	1	4			
12 Shouting			96	93	3	3			1	1	
13 Groaning			94	55	6	34		7		1	
14 Mumbling			89	83	10	11	1	3			
15 Complaining			96	93	4	4	3				
aPain Assessment in Impaired Cognition with 15 items, subdomain ranges of 0–15 and a total range of 0–45

Validity

Criterion validity

Correlations between PAIC15 consensus scores and the self-report pain scales of measurement 1 during rest and transfer are shown in Table 5. The PAIC15 had fair positive correlations with NRS, VAS, FPS, and the combined scale during rest (ranging from 0.35 with NRS to 0.50 with VAS). During transfer, the correlations between PAIC15 and the NRS, VAS and the combined scale were fair positive, varying from 0.38 (combined scale) to 0.43 (VAS). The PAIC15 correlated poorly with FPS (0.26).

Table 5 Correlation of PAIC15a consensus scoresb versus self-report pain scales total scores in patients with aphasia during rest and during transfer

Instrument	Spearman’s rho	PAIC15	FPS	NRS	VAS	Combination scale	
Rest	
 PAIC15	Correlation Coefficient

N

	1

71

	0.43**

45

	0.35*

45

	0.50**

44

	0.44**

45

	
 FPS	Correlation Coefficient

N

		1

45

	0.69**

45

	0.63**

44

	0.84**

45

	
 NRS	Correlation Coefficient

N

			1

45

	0.84**

44

	0.79**

45

	
 VAS	Correlation Coefficient

N

				1

44

	0.71**

44

	
Transfer	
 PAIC15	Correlation Coefficient

N

	1

70

	0.26

43

	0.40**

43

	0.43**

42

	0.38*

43

	
 FPS	Correlation Coefficient

N

		1

43

	0.73**

43

	0.87**

42

	0.92**

43

	
 NRS	Correlation Coefficient

N

			1

43

	0.84**

42

	0.81**

42

	
 VAS	Correlation Coefficient

N

				1

42

	0.92**

42

	
FPS Faces pain scale, possible range 0–10. NRS Numeric rating scale, possible range 0–10. VAS Visual analogue scale, possible range 0–10. Combination scale: possible range 0–10

*p < .050, ** p < .010

aPain Assessment in Impaired Cognition with 15 items, subdomain ranges of 0–15 and a total range of 0–45

bThe consensus scores of PAIC15 was based on consensus after discussing scores after independent observations during rest and transfer on day 1

Construct validity

For the construct validity, the results of the 3 hypotheses that were tested show that significant more pain was observed in persons with aphasia during transfer (median 3; IQR 2–6) than during rest (median 1; IQR 1–3); z = -4.15, p < .05.

Observations with the PAIC15 during rest showed more pain in persons with aphasia using pain medication (median 1.5; IQR 1-3.75) versus persons who use no pain medication (median 1; IQR 0–2). However, this difference was not significant, U (Nusing pain medication= 44, Nusing no pain medication= 26,) = 463, z = -1.36, p = .175. Similar results were found during transfer (with pain medication: median 3; IQR 2-6.75; without pain medication: median 3; IQR 2-5.5); U (Nusing pain medication= 44, Nusing no pain medication= 25,) = 487, z = − 0.80, p = .423. Our hypothesis was rejected.

During rest, less pain was observed in persons with joint diseases such as osteoarthritis or rheumatism (median 1; IQR 1-2.5) versus persons without these diseases (median 1; IQR 1–3). However, the difference was not significant; U (Nosteoarthritis/rheumatism= 9, Nno osteoarthritis/rheumatism= 55) = 238, z = − 0.19, p = .851. Similar results were found during transfer (with joint disease: median 2; IQR 1.5-6; without joint disease: median 3; IQR 2-6.25); U (Nosteoarthritis/rheumatism= 9, Nno osteoarthritis/rheumatism= 54) = 190, z = -1.05, p = .293. Our hypothesis was rejected.

Reliability

Internal consistency

The internal consistency of the PAIC15 was acceptable, varying between α = 0.73 and 0.93 during rest and between α = 0.82 and 0.85 during transfer. These values were assessed using the combined PAIC15 scores of observers A and B, during measurement 1, measurement 2 and measurement 1 and 2 together.

Intraobserver and interobserver agreement

Table 6 presents the intraobserver and interobserver agreement of the PAIC15 scores with the 4-point scale in persons with aphasia during rest and transfer. See Table 6 with percentages of ≥ 70% shaded- in green. Of the items in the domain facial expressions, all except ’opening mouth’ showed high intraobserver agreement (≥ 70%) during rest. During transfer, agreement was high only on the items ‘narrowing eyes’ and ‘raising upper lip’. Interobserver agreement was also high (≥ 70%) during rest. During transfer, only the items ‘narrowing eyes’ and ‘raising upper lip’ achieved high agreement (≥ 70%), as did intraobserver agreement. Of all items in the domains body movements and vocalizations, intra- and interobserver agreement was > 70% during rest and transfer. Percentage agreement was also assessed after dichotomization of the PAIC15 scores, indicating that pain related behaviours were either present (score 1–3) or absent (score 0). Intra- and interobserver agreement of the PAIC15 dichotomized scores are also presented in Table 6. This resulted in higher agreement percentages than when using the 4-point scale (Table 6). All dichotomized scores of the 15 items showed good reliability with percentages of 70 or higher for both intra- and interobserver agreement during rest and transfer.

Table 6 Intra- and interobserver agreement of the PAIC15a scores (in percentages) during rest and transfer in 71 patients with aphasia, both with 4-point and dichtomized sore

PAIC15 item	PAIC15 scores on the 4-point scale	PAIC15 dichotomized scores	
Intraobserver agreement
Percentage agreement	Interobserver agreement
Percentage agreement	Intraobserver agreement
Percentage agreement	Interobserver agreement
Percentage agreement	
Rest	Transfer	Rest	Transfer	Rest	Transfer	Rest	Transfer	
Facial expressions	
 1 Frowning	72	65	84	66	74	77	84	78	
 2 Narrowing eyes	91	82	96	80	92	86	96	84	
 3 Raising upper lip	94	79	96	72	94	82	96	76	
 4 Opening mouth	63	58	84	63	70	72	85	77	
 5 Looking tense	74	57	84	61	76	73	86	71	
Body movements	
 6 Freezing	95	82	98	76	96	85	99	78	
 7 Guarding	94	85	98	84	97	90	99	88	
 8 Resisting care	89	90	95	88	91	90	95	88	
 9 Rubbling	93	93	98	95	94	94	98	96	
 10 Restlessness	89	91	94	91	91	92	96	92	
Vocalizations	
 11 Using pain-related-words	97	80	99	90	97	82	99	94	
 12 Shouting	95	99	97	97	95	99	98	98	
 13 Groaning	93	71	95	76	94	78	96	82	
 14 Mumbling	93	84	93	86	91	85	94	88	
 15 Complaining	93	88	96	90	93	88	97	91	
aPain Assessment in Impaired Cognition with 15 items, subdomain ranges of 0–15 and a total range of 0–45

Discussion

This study aimed to examine the validity and reliability of the pain observation instrument PAIC15 in persons with aphasia and is therefore of clinical value for professionals to optimize pain assessment in persons with aphasia. Descriptive statistics of PAIC15 show that self-reporting pain was not possible in one third of participants (24/71). The prevalence of individual items of the PAIC15 observed in persons with aphasia was low for most items. Higher prevalence was observed in the facial expressions domain. This is in accordance with findings of a PAIC15 study in a long-term care setting in patients with dementia [33]. The items of the domains body movements and vocalizations showed the lowest prevalence. This result was expected, because of the minimal movement of the musculoskeletal system during rest. Regarding results during transfer, the overall prevalence of the individual items of PAIC15 was higher compared to the results during rest, which was expected.

Validity

The results of the current study indicate fair criterion validity because of largely fair positive correlations between PAIC15 and the self-report pain scales that could be completed by persons with aphasia. This study utilized consensus scores of PAIC15 after discussing the scores recorded by observer A and B following independent observations during rest and transfer on measurement 1. These consensus scores were needed to assess the correlations between the PAIC15 and self-report pain scales. If we compare the consensus scores to the scores of the independent observations, a few of the consensus scores were higher. However, discussion of the combined independent observations by observers A and B still yielded a higher score. An implication of this study is that using two observers improves the PAIC15 scores, because two observers see more than one observer during rest and transfer.

Another important finding, in terms of construct validity and assessed with hypothesis 1: significantly more pain was observed with the PAIC15 during transfer compared to during rest. However, hypothesis 2 (more pain observed when treated with pain medication compare to no-pain medication) was rejected. Contrary to studies of pain and pain medication in persons with dementia [26, 27], we did not find more pain in persons with aphasia when pain medication was used compared to when not treated with pain medication. Many studies have stressed that pain after stroke was under-recognized and persons received inadequate pain management [9, 34]. When pain is under-recognized and undertreated, while treatment would be effective, this hypothesis may not be suitable. Hypothesis 3 was also rejected because there was no difference in observed pain in persons with aphasia with and without joint disease. Joint disease is one of the most frequent general causes of pain, yet indeed, joint disease is not specific to stroke patients. Stroke patients experience significant pain after stroke, especially headache, shoulder pain, pain from increased muscle stiffness, and central post-stroke pain which are not related to joint disease and are uncommon in this study sample [35, 36]. Therefore, this hypothesis may not work well in this population and further research on causes of pain in stroke patients is warranted.

Reliability

Acceptable internal consistency of PAIC15 in persons with aphasia was examined. We found that intra- and interobserver agreement for the items of the PAIC15 domains body movements and vocalizations are both good (≥ 70%). Results on the domain facial expressions show good intraobserver agreement for almost all items and good interobserver for all items during rest. This is contrary to the findings during transfer with a high percentage only on both intra- and interobserver agreement for the items ‘narrowing eyes’ and ‘raising upper lip’. These results resemble those of Van Dalen-Kok et al. [33] who also found that fewer items in the domain facial expressions had good intraobserver- and interobserver agreement during both rest and transfer. Lower intra- and interobserver agreement for the facial expression items suggest that these items are more difficult to observe in a clinical setting. Research of Oosterman et al. [37] reported that recognizing and observing facial expressions for pain assessment in dementia requires specific training and education [37]. Assessing pain based on the observation of facial expressions in persons with dementia can be compared to persons with aphasia, because of their impaired cognition and communications problems. Percentages of 70 or higher for both intra- and interobserver agreement indicate good reliability of PAIC15 with dichotomized scores. This implies that assessing the presence of a pain-related item using PAIC15 is more reliable than assessing the degree/intensity of the pain-related items of PAIC15 with the 4-point scale in persons with aphasia.

Strength and limitations

Our study is the first to explore alternative methods for the long-standing and distressing situation of poor assessment and management of pain in persons with aphasia. Other strengths include the use of clinical situations and providing elaborate training for the research assistants. Also, no other pain research in persons with aphasia has used several self-report pain scales and a combined self-report scale. A limitation is that we did not check the competency of the different raters after training. However, we used a standardized training and each first observation of an observer was carried out with the researcher for instructions and practice in using PAIC15 independently. The prevalence of individual items observed in persons with aphasia was low for most items. The scores 2 and 3 of the PAIC15 were rarely rated, due to the fact that the observed persons with aphasia showed few items and the observers struggled to differentiate between score 2 or 3. Deciding between a 2 or 3 could be difficult; when is someone frowning with a ‘moderate’ or ‘great degree’? The rating of these scores has recently been revised and adjusted in the online PAIC15 e-learning.

It is also possible that the low scores are due to failure to observe behavior described in the PAIC15 items in persons with aphasia after stroke. This could lead to the question whether reporting items ‘not at all’ means that these persons do not experience any pain? This raises questions about the applicability of the PAIC15 in this population. However, literature reported that persons with aphasia can also experience pain, especially if they have communication problems. If this is the case and a self-report pain scale cannot be completed, pain may not be detected. The PAIC15 can meet this need by observing possible behaviors that indicate possible pain.

Another limitation might be that the time between the observations of measurements 1 and 2 varied from 1 to 7 days, and the use of self-report pain scales was not checked again after 7 days. Depending of the recovery of the stroke, it is recommended to check if self-reporting of pain is possible a week later. Within rehabilitation, spontaneous recovery can certainly occur within 7 days or the situation changes, e.g., re-admission to hospital or discharge home. These changes could affect the intraobserver agreement more strongly if the interval is 7 instead of 2 days.

Results may have been influenced because of current study was conducted during the COVID-19 pandemic. Which means that participants were observed during a period with restrictive rules to reduce the spread of COVID-19. These circumstances may potentially have influenced the observed behaviors of the participants using PAIC15, Future studies are recommended to determine if the results are valid in the post-COVID era.

Conclusions

Results show fair criterion validity, and significantly more pain was observed during transfer compared to rest using PAIC15 regarding construct validity. Regarding reliability, we found an acceptable internal consistency of PAIC15 and good intra- and interobserver agreement for most PAIC15 items, particularly for the domains body movements and vocalizations in persons with aphasia. This study shows that PAIC15 can be used to assess pain in persons with aphasia. Further research in the daily practice setting should clarify whether combining PAIC15 with self-report and other clinical leads will deliver results that can be confidently used in practice.

Supplementary Information

Supplementary Material 1.

Supplementary Material 2.

Supplementary Material 3.

Supplementary Material 4.

Abbreviations

PAIC15 Pain Assessment in Impaired Cognition

NRS Numerical Rating Scale

VAS Visual Analog Scale

FPS Faces Pain Scale

EU-COST European Cooperation in Science and Technology

UNC-ZH University Network for the Care sector – South Holland

COSMIN COnsensus based Standards for the selection of health Measurement INstruments

COVID-19 Coronavirus caused by severe acute respiratory syndrome -coronavirus -2  

WMO Medical Research Involving Human Subjects Act

Acknowledgements

The authors thank the persons with aphasia and their family members and/or legal representatives, speech and language therapists and nurses for their collaboration and participation.

Authors’ contributions

All authors have had substantial contributions to the design of the study. NJdV collected the data. NJdV, WA and HJAS analyzed and interpreted the data. NJdV, HJAS and WA drafted the manuscript. HJAS, JTvdS and WA reviewed the manuscript critically. All authors gave final approval of the manuscript to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Funding

This study was supported by the Zorgondersteuningsfonds with grant name PROM-6 and University Network for the Care sector South Holland (UNC-ZH).

Availability of data and materials

The datasets are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

The study protocol (P18.230) was reviewed by the Medical Ethics Review Committee Leiden-The Hague-Delft, and declared exempt from the Medical Research Involving Human Subjects Act (WMO). The study was conducted according to the guidelines of the Declaration of Helsinki. Informed consent was obtained using aphasia-friendly informed consent forms. When there was doubt about the ability to provide informed consent, informed consent was obtained from the legal representative.

Consent for publication

Informed consent for publication collected data was obtained from persons with aphasia using aphasia-friendly informed consent forms or from the legal representative.

Competing interests

The authors declare no competing interests.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Hjermstad MJ Studies comparing Numerical Rating scales, Verbal Rating scales, and Visual Analogue scales for Assessment of Pain intensity in adults: a systematic literature review J Pain Symptom Manag 2011 41 6 1073 93 10.1016/j.jpainsymman.2010.08.016
Hjermstad MJ, et al. Studies comparing Numerical Rating scales, Verbal Rating scales, and Visual Analogue scales for Assessment of Pain intensity in adults: a systematic literature review. J Pain Symptom Manag. 2011;41(6):1073–93.10.1016/j.jpainsymman.2010.08.016
2. Heller GZ Manuguerra M Chow R How to analyze the Visual Analogue Scale: myths, truths and clinical relevance Scandinavian J Pain 2016 13 67 75 10.1016/j.sjpain.2016.06.012
Heller GZ, Manuguerra M, Chow R. How to analyze the Visual Analogue Scale: myths, truths and clinical relevance. Scandinavian J Pain. 2016;13:67–75.10.1016/j.sjpain.2016.06.012
3. Kim EJ Buschmann MT Reliability and validity of the faces Pain Scale with older adults Int J Nurs Stud 2006 43 4 447 56 10.1016/j.ijnurstu.2006.01.001 16510146
Kim EJ, Buschmann MT. Reliability and validity of the faces Pain Scale with older adults. Int J Nurs Stud. 2006;43(4):447–56.16510146 10.1016/j.ijnurstu.2006.01.001
4. de Vries NJ Sloot PH Achterberg WP Pain and pain assessment in stroke patients with aphasia: a systematic review Aphasiology 2017 31 6 703 19 10.1080/02687038.2016.1254150
de Vries NJ, Sloot PH, Achterberg WP. Pain and pain assessment in stroke patients with aphasia: a systematic review. Aphasiology. 2017;31(6):703–19.10.1080/02687038.2016.1254150
5. Benaim C Use of the faces Pain Scale by left and right hemispheric stroke patients Pain 2007 128 1–2 52 8 10.1016/j.pain.2006.08.029 17027154
Benaim C, et al. Use of the faces Pain Scale by left and right hemispheric stroke patients. Pain. 2007;128(1–2):52–8.17027154 10.1016/j.pain.2006.08.029
6. de Vries NJ Sloot PH Achterberg WP Pain and pain assessment in stroke patients with aphasia: a systematic review Aphasiology 2016 31 6 703 19 10.1080/02687038.2016.1254150
de Vries NJ, Sloot PH, Achterberg WP. Pain and pain assessment in stroke patients with aphasia: a systematic review. Aphasiology. 2016;31(6):703–19.10.1080/02687038.2016.1254150
7. Smith JH Inability to self-report pain after a stroke: a population-based study Pain 2013 154 8 1281 6 10.1016/j.pain.2013.04.006 23725781
Smith JH, et al. Inability to self-report pain after a stroke: a population-based study. Pain. 2013;154(8):1281–6.23725781 10.1016/j.pain.2013.04.006
8. Mandysova P Assessment instruments used for self-report of pain in hospitalized stroke patients with communication problems: a scoping review Jbi Evid Synthesis 2022 20 6 1511 36 10.11124/JBIES-21-00047
Mandysova P, et al. Assessment instruments used for self-report of pain in hospitalized stroke patients with communication problems: a scoping review. Jbi Evid Synthesis. 2022;20(6):1511–36.10.11124/JBIES-21-00047
9. Harrison RA Field TS Post stroke pain: identification, assessment, and therapy Cerebrovasc Dis 2015 39 3–4 190 201 10.1159/000375397 25766121
Harrison RA, Field TS. Post stroke pain: identification, assessment, and therapy. Cerebrovasc Dis. 2015;39(3–4):190–201.25766121 10.1159/000375397
10. Price CIM Curless RH Rodgers H Can stroke patients use visual analogue scales? Stroke 1999 30 7 1357 61 10.1161/01.STR.30.7.1357 10390307
Price CIM, Curless RH, Rodgers H. Can stroke patients use visual analogue scales? Stroke. 1999;30(7):1357–61.10390307 10.1161/01.STR.30.7.1357
11. Husebo BS Achterberg W Flo E Identifying and managing Pain in people with Alzheimer’s Disease and other types of dementia: a systematic review CNS Drugs 2016 30 6 481 97 10.1007/s40263-016-0342-7 27240869
Husebo BS, Achterberg W, Flo E. Identifying and managing Pain in people with Alzheimer’s Disease and other types of dementia: a systematic review. CNS Drugs. 2016;30(6):481–97.27240869 10.1007/s40263-016-0342-7
12. de Vries NJ Measuring Pain in Aphasia: validity and reliability of the PACSLAC-D Pain Manag Nurs 2023 24 4 e68 74 10.1016/j.pmn.2023.03.010 37100703
de Vries NJ, et al. Measuring Pain in Aphasia: validity and reliability of the PACSLAC-D. Pain Manag Nurs. 2023;24(4):e68–74.37100703 10.1016/j.pmn.2023.03.010
13. Herr K Pain assessment in the patient unable to self-report: position statement with clinical practice recommendations Pain Manag Nurs 2011 12 4 230 50 10.1016/j.pmn.2011.10.002 22117755
Herr K, et al. Pain assessment in the patient unable to self-report: position statement with clinical practice recommendations. Pain Manag Nurs. 2011;12(4):230–50.22117755 10.1016/j.pmn.2011.10.002
14. Lautenbacher S, Walz AL, Kunz M. Using observational facial descriptors to infer pain in persons with and without dementia. BMC Geriatr, 2018. 18.
15. Kunz M The Pain Assessment in impaired cognition scale (PAIC15): a multidisciplinary and international approach to develop and test a meta-tool for pain assessment in impaired cognition, especially dementia Eur J Pain 2020 24 1 192 208 10.1002/ejp.1477 31487411
Kunz M, et al. The Pain Assessment in impaired cognition scale (PAIC15): a multidisciplinary and international approach to develop and test a meta-tool for pain assessment in impaired cognition, especially dementia. Eur J Pain. 2020;24(1):192–208.31487411 10.1002/ejp.1477
16. van Dalen-Kok AH Pain Assessment in impaired cognition (PAIC): content validity of the Dutch version of a new and universal tool to measure pain in dementia Clin Interv Aging 2018 13 25 34 10.2147/CIA.S144651 29317807
van Dalen-Kok AH, et al. Pain Assessment in impaired cognition (PAIC): content validity of the Dutch version of a new and universal tool to measure pain in dementia. Clin Interv Aging. 2018;13:25–34.29317807 10.2147/CIA.S144651
17. El Hachioui H The ScreeLing: occurrence of linguistic deficits in acute aphasia post-stroke J Rehabil Med 2012 44 5 429 35 10.2340/16501977-0955 22549651
El Hachioui H, et al. The ScreeLing: occurrence of linguistic deficits in acute aphasia post-stroke. J Rehabil Med. 2012;44(5):429–35.22549651 10.2340/16501977-0955
18. Doesborgh SJ Linguistic deficits in the acute phase of stroke J Neurol 2003 250 8 977 82 10.1007/s00415-003-1134-9 12928919
Doesborgh SJ, et al. Linguistic deficits in the acute phase of stroke. J Neurol. 2003;250(8):977–82.12928919 10.1007/s00415-003-1134-9
19. de Waal MWM Observational pain assessment in older persons with dementia in four countries: Observer agreement of items and factor structure of the Pain Assessment in impaired cognition Eur J Pain 2020 24 2 279 96 10.1002/ejp.1484 31520424
de Waal MWM, et al. Observational pain assessment in older persons with dementia in four countries: Observer agreement of items and factor structure of the Pain Assessment in impaired cognition. Eur J Pain. 2020;24(2):279–96.31520424 10.1002/ejp.1484
20. van der Steen JT et al. Probable Pain on the Pain Assessment in Impaired Cognition (PAIC15) Instrument: Assessing Sensitivity and Specificity of Cut-Offs against Three Standards. Brain Sci, 2021. 11(7).
21. De Vet H, Terwee CB, Mokkink LB, Knol DL. Measurement in Medicine. A Practical Guide., ed. P.g.t.B.a. Epidemiology. 2011, Cambridge: Cambridge University Press.
22. Mokkink LB The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes J Clin Epidemiol 2010 63 7 737 45 10.1016/j.jclinepi.2010.02.006 20494804
Mokkink LB, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–45.20494804 10.1016/j.jclinepi.2010.02.006
23. Hadjistavropoulos T Pain assessment in elderly adults with dementia Lancet Neurol 2014 13 12 1216 27 10.1016/S1474-4422(14)70103-6 25453461
Hadjistavropoulos T, et al. Pain assessment in elderly adults with dementia. Lancet Neurol. 2014;13(12):1216–27.25453461 10.1016/S1474-4422(14)70103-6
24. Lints-Martindale AC A comparative investigation of Observational Pain Assessment Tools for older adults with dementia Clin J Pain 2012 28 3 226 37 10.1097/AJP.0b013e3182290d90 21904200
Lints-Martindale AC, et al. A comparative investigation of Observational Pain Assessment Tools for older adults with dementia. Clin J Pain. 2012;28(3):226–37.21904200 10.1097/AJP.0b013e3182290d90
25. Zwakhalen SM Pain in elderly people with severe dementia: a systematic review of behavioural pain assessment tools BMC Geriatr 2006 6 3 10.1186/1471-2318-6-3 16441889
Zwakhalen SM, et al. Pain in elderly people with severe dementia: a systematic review of behavioural pain assessment tools. BMC Geriatr. 2006;6:3.16441889 10.1186/1471-2318-6-3
26. Rajkumar AP Epidemiology of Pain in people with dementia living in Care homes: Longitudinal Course, Prevalence, and treatment implications J Am Med Dir Assoc 2017 18 5 e4531 6 10.1016/j.jamda.2017.01.024
Rajkumar AP, et al. Epidemiology of Pain in people with dementia living in Care homes: Longitudinal Course, Prevalence, and treatment implications. J Am Med Dir Assoc. 2017;18(5):e4531–6.10.1016/j.jamda.2017.01.024
27. van Dam PH Quality of Life and Pain Medication use in persons with Advanced Dementia Living in Long-Term Care facilities J Am Med Dir Assoc 2019 20 11 1432 7 10.1016/j.jamda.2019.02.019 30982716
van Dam PH, et al. Quality of Life and Pain Medication use in persons with Advanced Dementia Living in Long-Term Care facilities. J Am Med Dir Assoc. 2019;20(11):1432–7.30982716 10.1016/j.jamda.2019.02.019
28. Arendt-Nielsen L Joint pain: more to it than just structural damage? Pain 2017 158 Suppl 1 S66 73 10.1097/j.pain.0000000000000812 28151834
Arendt-Nielsen L. Joint pain: more to it than just structural damage? Pain. 2017;158(Suppl 1):S66–73.28151834 10.1097/j.pain.0000000000000812
29. Breivik H Survey of chronic pain in Europe: prevalence, impact on daily life, and treatment Eur J Pain 2006 10 4 287 333 10.1016/j.ejpain.2005.06.009 16095934
Breivik H, et al. Survey of chronic pain in Europe: prevalence, impact on daily life, and treatment. Eur J Pain. 2006;10(4):287–333.16095934 10.1016/j.ejpain.2005.06.009
30. Ebell MH Osteoarthritis: Rapid evidence review Am Fam Physician 2018 97 8 523 6 29671497
Ebell MH. Osteoarthritis: Rapid evidence review. Am Fam Physician. 2018;97(8):523–6.29671497
31. de Vet HC When to use agreement versus reliability measures J Clin Epidemiol 2006 59 10 1033 9 10.1016/j.jclinepi.2005.10.015 16980142
de Vet HC, et al. When to use agreement versus reliability measures. J Clin Epidemiol. 2006;59(10):1033–9.16980142 10.1016/j.jclinepi.2005.10.015
32. de Vet HC Clinicians are right not to like Cohen’s kappa BMJ 2013 346 f2125 10.1136/bmj.f2125 23585065
de Vet HC, et al. Clinicians are right not to like Cohen’s kappa. BMJ. 2013;346:f2125.23585065 10.1136/bmj.f2125
33. van Dalen-Kok AH Pain assessment in impaired cognition: observer agreement in a long-term care setting in patients with dementia Pain Manag 2019 9 5 461 73 10.2217/pmt-2019-0025 31403394
van Dalen-Kok AH, et al. Pain assessment in impaired cognition: observer agreement in a long-term care setting in patients with dementia. Pain Manag. 2019;9(5):461–73.31403394 10.2217/pmt-2019-0025
34. Widar M Long-term pain conditions after a stroke J Rehabil Med 2002 34 4 165 70 10.1080/16501970213237 12201611
Widar M, et al. Long-term pain conditions after a stroke. J Rehabil Med. 2002;34(4):165–70.12201611 10.1080/16501970213237
35. Trouvin AP Perrot S Pain in osteoarthritis. Implications for optimal management Joint Bone Spine 2018 85 4 429 34 10.1016/j.jbspin.2017.08.002 28889010
Trouvin AP, Perrot S. Pain in osteoarthritis. Implications for optimal management. Joint Bone Spine. 2018;85(4):429–34.28889010 10.1016/j.jbspin.2017.08.002
36. Liampas A Prevalence and Management challenges in Central Post-stroke Neuropathic Pain: a systematic review and Meta-analysis Adv Therapy 2020 37 7 3278 91 10.1007/s12325-020-01388-w
Liampas A, et al. Prevalence and Management challenges in Central Post-stroke Neuropathic Pain: a systematic review and Meta-analysis. Adv Therapy. 2020;37(7):3278–91.10.1007/s12325-020-01388-w
37. Oosterman JM The use of facial expressions for pain assessment purposes in dementia: a narrative review Neurodegener Dis Manag 2016 6 2 119 31 10.2217/nmt-2015-0006 27032976
Oosterman JM, et al. The use of facial expressions for pain assessment purposes in dementia: a narrative review. Neurodegener Dis Manag. 2016;6(2):119–31.27032976 10.2217/nmt-2015-0006
