
==== Front
Health Res Policy Syst
Health Res Policy Syst
Health Research Policy and Systems
1478-4505
BioMed Central London

1152
10.1186/s12961-024-01152-4
Review
Citizen involvement in research on technological innovations for health, care or well-being: a scoping review
http://orcid.org/0000-0002-1003-0794
van Leersum Catharina Margaretha c.m.vanleersum@utwente.nl

12
Jaschinski Christina 3
Bults Marloes 3
van der Zwart Johan 1
1 https://ror.org/006hf6230 grid.6214.1 0000 0004 0399 8953 Department of Science, Technology, and Policy Studies, Faculty of Behavioural, Management, and Social Sciences, University of Twente, Enschede, The Netherlands
2 grid.36120.36 0000 0004 0501 5439 Faculty of Humanities, Open University, Heerlen, The Netherlands
3 https://ror.org/005t9n460 grid.29742.3a 0000 0004 5898 1171 Technology, Health, and Care Research Group, Saxion University of Applied Sciences, Enschede, The Netherlands
2 9 2024
2 9 2024
2024
22 1199 11 2022
20 5 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by/4.0/ Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Citizen science can be a powerful approach to foster the successful implementation of technological innovations in health, care or well-being. Involving experience experts as co-researchers or co-designers of technological innovations facilitates mutual learning, community building, and empowerment. By utilizing the expert knowledge of the intended users, innovations have a better chance to get adopted and solve complex health-related problems. As citizen science is still a relatively new practice for health and well-being, little is known about effective methods and guidelines for successful collaboration. This scoping review aims to provide insight in (1) the levels of citizen involvement in current research on technological innovations for health, care or well-being, (2) the used participatory methodologies, and (3) lesson’s learned by the researchers.

A scoping review was conducted and reported in accordance with the PRISMA-ScR guidelines. The search was performed in SCOPUS in January 2021 and included peer-reviewed journal and conference papers published between 2016 and 2020. The final selection (N = 83) was limited to empirical studies that had a clear focus on technological innovations for health, care or well-being and involved citizens at the level of collaboration or higher. Our results show a growing interest in citizens science as an inclusive research approach. Citizens are predominantly involved in the design phase of innovations and less in the preparation, data-analyses or reporting phase. Eight records had citizens in the lead in one of the research phases.

Researcher use different terms to describe their methodological approach including participatory design, co-design, community based participatory research, co-creation, public and patient involvement, partcipatory action research, user-centred design and citizen science. Our selection of cases shows that succesful citizen science projects develop a structural and longitudinal partnership with their collaborators, use a situated and adaptive research approach, and have researchers that are willing to abandon traditional power dynamics and engage in a mutual learning experience.

Keywords

Citizen science
Healthcare
Participatory research
Co-creation
Innovation
eHealth
Co-researchers
Well-being
issue-copyright-statement© BioMed Central Ltd., part of Springer Nature 2024
==== Body
pmcBackground

Many technologies are developed to address health, care or well-being related challenges, such as mental health, manage health at home, or coping with an understaffed healthcare sector [1]. These technologies can include new diagnostic or therapeutic methods and devices, ambient assisted living technologies, and eHealth such as monitoring, regulatory, or advisory apps. There is a lot of research into the design, development, implementation, and evaluation of new technologies. However, the scoping review of Krick et al. showed that implementation of new technologies remains a challenge due to lack of in-depth knowledge about the care environment or users [2].

Citizen science, or the use of scientific principles and methods by non-professional scientists, may be a powerful method to improve public participation in research as well as public health and is regarded as a promising research approach to improve implementation and adoption of technological developments in health, care or well-being by collaborating with the intended users as experience experts [3]. The initial drivers to develop citizen science methods were the increase of research capacity [4], and the possibility to respond more effectively on complex societal problems due to the addition of lay, local or traditional knowledge to scientific knowledge [5, 6]. Besides the positive contributions of citizen science to the scientific processes or knowledge, it has potential advantages for lay people being co-researchers of a project. In health related research as co-researcher could result in a broader understanding of their own health and well-being and acquisition of knowledge about a healthy lifestyle [7]. A list of 10 benefits for co-researchers was developed by Haywood [8]. These benefits include enhancing science knowledge and literacy, enhancing understanding of scientific methods, improving access to scientific information, increasing scientific thinking, improving interpretation skills, diminishing the gap between science and people, strengthening connections between people and environment, empowering co-researchers, increasing community-building, changing attitudes, influencing policy, and gaining access to broader networks [8]. King et al. showed similar benefits in several citizen science case studies, where the main benefits concerned scientific literacy [9].

Currently, there is no uniform/ broadly accepted definition of citizen science. So far there is no consensus, and it is even discussed whether a definition is necessary [10, 11]. This non-existence of a definition is accepted and allows stakeholders working in citizen science to use different terms and methodologies [12]. In this paper we define citizen science as research (partly) executed by non-professional scientists, volunteers, citizen scientists or co-researchers. Often the citizen scientists work together or are guided by academic researchers or governmental organisations. Thus, a citizen should be taking active part in any of the research activities and ideally collaborate with professional researchers. We take a similar perspective as Heigl et al. and exclude projects from our definition that use participants only at as providers and resources of knowledge without any active involvement in scientific activities [11].

Citizen science is an approach which comprises a range of participatory approaches that are embraced as collaborative practices and show promising results [3]. Although citizen science in the healthcare domain is a relatively new and rare phenomenon, it is already an established approach in research fields such as ecology, conservation, and biology [5, 13]. Projects with citizen science approaches attract hundreds of thousands of participants, who are involved in different tasks varying in complexity [14]. The inclusion of citizens is valuable for science [15], for example, when the expertise of the general public or a specific population of citizens is needed to understand and solve a problem [16]. The collective intelligence of the citizens has led to scientific discoveries such as protein folding [17] and air pollution [18]. However, citizens science projects also deal with challenges when it comes to the selection of participating citizens, the needed and available competences of citizens, and the credibility of knowledge gathered by or with the citizens [5, 19].

TOPFIT Citizenlab is a three-year research and innovation program aiming at increasing citizen involvement and participation in researching, testing, modifying and implementing technological innovations for prevention and health promotion. One of the objectives withing Citizenlab is to explore and understand useful methodologies of citizen science in health and well-being research. There is limited knowledge on the benefits of citizen science, and the normative assumptions are often taken for granted [20, 21]. Examples of citizen science do not show a thorough analysis of successes and failures, therefor, little is understood about the advantages or pitfalls [21]. In the field of citizen science, to our knowledge nine literature reviews are performed by Carpini et al., Irvin and Stansbury, McGuire, Conrad and Hilchey, Garau, Kimura and Kinchy, Kullenberg and Kasperowski, Peter et al., and Ianiello et al. [13, 21–28]. These reviews focus on the use of citizen science to encourage public discourse, public decision-making, public management, or environmental research. To our knowledge, there were two literature reviews on citizen science with a focus on health, care or well-being published by Domecq et al. and Malterud and Elvbakken [29, 30]. These review focus on patient engagement in healthcare research and show that it is used in many settings. Often the research methods are adapted to make engagement possible for the citizens, and patient engagement is low during the data collection phases or rather tokenistic. Authors of both reviews argue that further research is needed to determine which valuable collaborative methods can achieve more active involvement of patients and define ideas and strategies for citizen science in health research [29, 30].

In our scoping review we analysed the citizen science methodologies applied in technology development for health, care or well-being. We were specifically interested in which innovation phases citizens were involved, and in the level of citizen involvement. The aim was to provide insight in (1) the levels of citizen involvement in current research on technological innovations for health, care or well-being, (2) the used participatory methodologies, and (3) lesson’s learned by the researchers.

Research phases and citizen involvement

To reach the aim, we started by distinguishing three research phases, namely: (1) preparation of research, which included the design of a research, the recruitment process, and design of materials such as interview guides, (2) data collection, which included all (iterative) steps of technology design, data collection and use of feedback throughout these iterations, and (3) data analysis and evaluation, which included methods of analysis as well as reporting of research data and outcome. With this distinction it is possible that one record might describe one, two or three phases in which citizens were involved.

Second, we assessed the level of citizen involvement. We proposed levels of citizen involvement based on the participation ladder of Arnstein and citizen science terminology defined by Hakley [31, 32]. Arnstein’s participation ladder is more used in citizen science research to discuss the methodology and possibilities of citizen involvement, for example by van Leersum et al. [2020] about the involvement of citizens in the need of long-term care during the development of a tool for self-assessment, and by Kotus to explore the level of participation of citizens in the urban policy making of a Polish city [33, 34]. The ladder consists of eight steps including manipulation, therapy, informing, consultation, placation, partnership, delegated power, and citizen control [31]. Each step on the ladder characterizes the role of the participant and the researcher. The higher participants are placed on the ladder, the more influence they will have. This ladder configuration shows that one level is building on the previous level, but there is no logical progression from one to another [35]. The steps manipulation and therapy are considered non-participation. The steps of informing, consultation, and placation are considered tokenism, in which the citizens have superficial involvement. From the informing step in which the citizens are informing the researchers about their personal situation or experiences, towards the placation step in which citizens advise the researchers about needed adjustment, research or innovations. The highest steps of partnership, delegated power, and citizen control are called citizen power, in which the citizens have more and more in-depth control and influence. In partnership the relation and tasks of the citizens and researchers are equal, and with citizen control, the citizens lead the research and researchers are available to support [31].

Hakley uses four different terms to distinguish involvement in citizen science research [32]. First, crowdsourcing, in which cognitive abilities of citizens are not needed and the citizens use digital devices to collect data that are automatically send and analysed by researchers at a later stage. Second, distributed intelligence, in which the citizens get a certain task such as interpretation or classification of data. Third, participatory science, in which citizens are formulating research questions or problems, and are participating in data collection, but analysis is often done by the researchers. Fourth and last, collaborative science, also referred to extreme citizen science, in which a project is carried out by citizens without researchers being actively involved [32]. Although it might seem from terminology and categorisations of citizen involvement that the higher the level of involvement the better, we will not make this argument in this scoping review. All forms of citizen science could be valuable and often citizen science research could not be defined to one level alone [32, 36]. It should be up to the team “to determine the best design specifications for their own unique context, enabling citizen science to achieve its full potential” [10]. Also, as Ferro and Molinari [2009] argue, researchers need to be aware of the complexity of citizens [37]. There is a considerable variation across the population of citizens who might be interested to be involved in different forms. This variation could be in terms of interest as well as educational level or technological skills. For some citizens, these differences make it more difficult or less desirable to participate on higher steps of the ladder. Researchers should be more aware of the variation and should know when to make a step towards the citizens rather than to expect the citizens to move towards them [37].

In our analysis we combine the terminology as used by Arnstein (1969) and Hakley (2013) and define four levels of citizen involvement [31, 32]. The first level is the level of non-involvement, this level includes manipulation, therapy, and crowdsourcing. The second level is sharing knowledge, including informing, consultation, placation, and distributed intelligence. The third level is collaboration, which includes partnership, delegated power, and participatory science, and the fourth level is citizens in the lead, including citizen control, and collaborative science. These four levels of citizen involvement were considered in view of the different research phases. Based on the scoping review, we highlighted five cases, and explored the variation in citizens, how they are involved, and what worked well or what was experienced problematic.

Method

Methodological approach

A scoping review, following PRISMA-ScR guidelines, was performed to examine how citizens are involved in conducted research in the field of health, care or well-being [38]. Scoping review is a reviewing method to map a research area and identify all relevant literature regardless of the study design. This approach is developed to contextualize knowledge and identify the current state of understanding of a specific topic, and to assists in identifying the aspects which are investigated and the aspects which are less or not investigated [39]. A scoping review was chosen due to the broad topic of citizen science and the application of many different study designs within this topic [39]. We investigated citizen science methodologies applied in innovation processes for health, care or well-being. In this article we provide insight in (1) the levels of citizen involvement in current research on technological innovations for health, care or well-being, (2) the used participatory methodologies, and 3) lesson’s learned by the researchers as described in identified records.

Search strategy

First, we developed a search string that included many combinations of search terms within four research fields: ‘how’, ‘why’, ‘what’, and ‘who’ (Table 1). How considers the involvement of citizens, why is health, care or well-being related, what contains the technological innovations, and who are the citizens. For each of these fields specific search terms were defined. For example, the search string for how:Table 1 The main fields of the search string and all search term

How	Involvement	
	Citizen science; Participatory research; User involvement; Citizen engagement; Citizen knowledge; Patient and public involvement; Community-based participatory research; Community engagement; Community participation; Citizen participation; Co-creation; Co-design; Participatory design; Co-production; User-centred design	
Why	Health and well-being	
	Health; Healthcare; Care; Wellbeing; Well-being; Lifestyle; Healthy	
What	Technology innovation	
	Innovation; Technology; App; Digital; Robotics; Robots; Smart; Gaming; Web based; Games; Game design; Tool; Computer-based; Ehealth, E-health; Telehealth; M-health; Mhealth; Mobile health; E-therapy; Digital health	
Who	Citizens	
	Citizen; Public; Community; Patients; Clients; Elderly; Seniors; Caregivers; Neighbours; Family; Children; Parents; Adults	

TITLE-ABS-KEY (“citizen science” OR “participatory research” OR “user involvement” OR “citizen engagement” OR “citizen knowledge” OR “patient and public involvement” OR “community-based participatory research” OR “community engagement” OR “community participation” OR “citizen participation” OR “user-centred design” OR “co-creation” OR “co-design” OR “participatory design” OR “co-production”)

As proposed by Jackson and Waters, different combinations of search terms were tested and discussed by the research team and an information specialist [40]. The records acquired with each combination were compared and based on the included and excluded records by addition or removal of search terms, the final search terms were chosen. The final search string was formed by using AND in between the how, why, what, and who, and the terms within each part were connected using OR. To conduct the search, the final search string was entered in January 2021 into the database SCOPUS. The same information specialist that assisted the research team in testing and finalizing the search string, recommended the use of SCOPUS for its suitability to our scoping review's wide-ranging interest and aim. SCOPUS, being a large international and multi-disciplinary database of peer-reviewed literature, such as scientific journals, books, and conference proceedings, and indexes a broad spectrum of studies in the field of science, technology, medicine, social sciences, and arts and humanities, aligning closely with the scope of our review.

Selection of records

For the selection of the records several inclusion criteria and restrictions were chosen. The inclusion criteria were (1) empirical studies, (2) studies describing a method in which citizens were actively involved, (3) health related topics, and (4) technological innovation. Published records were not eligible if:The record did not present the execution of an empirical study. Records describing exclusively the methodology, lessons learned, study design, or reviews are excluded.

The used method did not actively involve citizens. Citizens had to be involved to a larger extent than 'just' as subjects. Citizens could be professionals if they are the end-users of the technological innovation.

The study did not aim to improve health, care or wellbeing.

Technological innovation was not part of the record. We used the definition of Behney “Medical technologies include the drugs, devices, and medical and surgical procedures used in medical care, and the organizational and supportive systems within which such care is provided” [41].

Furthermore, we restricted our search to records that were published in the last five years, were peer reviewed or conference papers, and published in English. There were no restrictions on type of research or study design.

The first 200 titles and abstracts were screened by the four researchers. The exclusion criteria for the first screening consisted of no citizen involvement, no health, no technology, and not empirical. The four researchers discussed the differences of inclusion after screening the first 100 records. Then repeated the process for the next 100 records. The remaining records were split and both sets were screened by two researchers, each researcher couple discussed their differences. In case of doubt, the titles and abstracts were also discussed with the other researcher couple.

Ranking citizen involvement

To classify the used methodologies of the included records, we developed a matrix to rank the citizen involvement within the described research phases. The citizen involvement levels were based on the participation ladder of Arnstein and the citizen science terminology of Hakley [31, 32]. We defined four levels:Non-involvement: there is no active involvement of citizens.

Sharing knowledge: the citizen receives information from the researcher, the citizen provides (advisory) information to the researcher, there is a dialogue in which decisions were made by the researcher.

Collaboration: the citizen and researcher are partners, equal in dialogue and equal in taking decisions, the researcher assists in all research activities.

Citizens in the lead: the citizen is leading and taking decisions, very minimal assistance of the researcher is provided, only on request of the citizen.

The methods sections of the full text records were screened and ranked on research phase and level of citizen involvement. The first 20 full text rankings were performed by four researchers. A record was directly excluded when it did not fit anywhere in the matrix. The four researchers discussed the differences of inclusion and ranking after reading the first 20 full texts. Then repeated this process for the next 10. The remaining records were divided among the researchers. In case of doubt, another researcher read the full text as well to align the analysis.

Data analysis

The research phases and the level of citizen involvement was screened. The analysis was based on the descriptions of the methodology in the records. We decided to take only records ranked on the levels of collaboration or citizens in the lead in the final steps of the analysis. The following information was extracted from the records: authors, year of publication, country of research, used methodology, health, care or well-being research topic, technological innovation, number and characteristics of involved citizens, and level of citizen involvement. Besides these details, five records were described in more detail as example cases. The presented scoping review was performed and reported on in compliance with the PRISMA-ScR guidelines.

Results

Selection and characteristics of records

In total 3846 records were retrieved (Fig. 1), no duplicates were found. After screening the titles and abstracts of the records, 2983 records were removed based on the inclusion criteria: empirical study, citizen involvement, focus on health, care or well-being, and technological innovation. Full texts were obtained, and the methods sections of the remaining 861 records were screened and ranked. During methods screening and full text reading 667 records were removed due to no citizen involvement. From the 194 remaining record, 110 records were put-aside based on citizen involvement solely on the levels of sharing knowledge or non-involvement. A total of 83 records were included for further analysis and are presented in Table 2. The research team used the Population, Context, and Concepts (PCC) criteria to describe characteristics of the included studies. These criteria are utilized in the following text and example cases, as well as in Table 2. More specifically, Table 2 displays, for each record, the year of publication, the country where the study was conducted, the terminology used for research design, the health, care, or well-being topic, technological innovations, the involved citizens, and the level of involvement during various research phases.Fig. 1 The search, reported following PRISMA-ScR guidelines, delivered 3846 records with no duplicates. After title and abstract screening 2983 records were excluded. Method screening resulted in the exclusion of 667 records. In 111 records, the study solely involved citizens on the level of sharing knowledge or non-involvement, therefore, 83 records were finally included

Table 2 All 83 included records in alphabetical order, reported following PRISMA-ScR guidelines

Authors	Year of publication	Country	Used terminology to describe Method	Health, care or well-being topic	Technological innovation	Involved citizens	Level of involvement in preparation	Level of involvement in data collection	Level of involvement in evaluation	
Abbass-Dick et al. [42]	2018	Canada	Participatory design	Breastfeeding	eHealth	11 Indigenous mothers, 9 care providers	Non-involvement	Collaboration	Non-involvement	
Abbass-Dick et al. [43]	2020	Canada	Co-creation	Breastfeeding	eHealth	12 young mothers, 8 health care providers	Non-involvement	Collaboration	Non-involvement	
Ahmed et al. [44]	2019	United States	Participatory design	Cardiac resynchronization therapy	Electronic health data	24 adult patients	Collaboration	Collaboration	Non-involvement	
Allin et al. [45]	2018	Canada	Participatory design	Spinal cord injury	Online self-management tool	9 persons with SCI; co-designers; key informants	Collaboration	Collaboration	Sharing knowledge	
Arevian et al. [46]	2018	United States	Co-creation	Community resilience	Mobile application	30 people from the community; co-leadership	Sharing knowledge	Collaboration	Collaboration	
Arevian et al. [47]	2020	United States	Usability study	Obsessive compulsive disorder	Mobile texting apps	28 patients, 4 care providers	Non-involvement	Collaboration	Sharing knowledge	
Ceasar et al. [48]	2017	United States	Community based participatory research	Community based health	mHealth technology	159 people from African American communities	Collaboration	Collaboration	Collaboration	
De Cocker et al. [49]	2019	Belgium	Citizen Science	Public health guidelines	Online survey	6246 adults	Sharing knowledge	Collaboration	Sharing knowledge	
DeRenzi et al. [50]	2017	India	Participatory design	Community health	Web- and voice-based system	71 community health workers	Non-involvement	Sharing knowledge	Collaboration	
Dewa et al. [51]	2020	United Kingdom	Co-production	Mental health difficulties	Technology to detect mental health deterioration	7 young people; co-researchers; advisory group	Collaboration	Collaboration	Collaboration	
Doty et al. [52]	2020	United States	Community based participatory research	Parenting programmes for Latinx families	Mobile app/mHealth	115 Latinx parents

Latinx-serving organisations Latinx community facilitators

	Collaboration	Collaboration	Collaboration	
Edwards et al. [53]	2020	United States & Ireland	Community based participatory research	Home-based spirometry in pulmonary fibrosis	Mobile app/mHealth	36 patients	Collaboration	Non-involvement	Non-involvement	
Elk et al. [54]	2020	United States	Community based participatory research	Palliative care	Telehealth	14 community members with different ethnicities	Collaboration	Collaboration	Collaboration	
Erlingsdóttir et al. [55]	2019	Sweden	Co-design	Support for home care staff	eHealth services	16 home care staff from 4 municipalities	Non-involvement	Collaboration	Collaboration	
Evans et al. [56]	2016	United Kingdom	Community based participatory research	HIV testing	mHealth	12 community members	Collaboration	Collaboration	Collaboration	
Ferguson et al. [57]	2018	United Kingdom	Participatory design	Educational programme for hearing aid users	Interactive video tutorials	3 hearing aid users and 1 charity advocate people with hearing loss, panel of 11 audiologists	Non-involvement	Collaboration	Sharing knowledge	
Ferri et al. [58]	2020	Italy	Co-creation	Cyber bullying	Massive open online course (MOOC)	30 children, 2 teachers; co-creators	Non-involvement	Collaboration	Sharing knowledge	
Fletcher and Mullett [59]	2016	Canada	Participatory action research	Healthy lifestyle	Digital stories	230 younger adults, 14 older adults; development team	Collaboration	Citizens in the lead	Collaboration	
Franco et al. [60]	2016	United States	Co-design	Post-Traumatic Stress Disorder	mHealth/Mobile app	32 veterans (peer mentors and mentees)	Collaboration	Collaboration	Collaboration	
Frauenberger et al. [61]	2020	Austria	Participatory design	Social Play for mixed groups of neurodivergent and neurotypical children	Social Play Technologies	16 neurodivergent and neurotypical children aged 7 to 12, 1 teacher	Collaboration	Citizens in the lead	Sharing knowledge	
Garcia et al. [62]	2019	Brazil	Co-creation	Alcohol and drugs rehabilitation	Digital games	10 adults with alcohol/drug addiction, 2 healthcare professionals	Sharing knowledge	Citizens in the lead	Collaboration	
Gardsten [63]	2017	Sweden	Participatory design	Self-management of Type 2 diabetes	ICT self-management service	5 adults with type 2 diabetes, 2 diabetes nurses	Collaboration	Collaboration	Sharing knowledge	
Garmendia et al. [64]	2019	Spain	Co-design	Oncology follow-up	Visualization tool	2 oncologists and 2 nurses	Non-involvement	Collaboration	Non-involvement	
Giroux et al. [65]	2019	Canada	Co-design	Supporting caregivers of older adults in home care	eHealth	74 caregivers, community workers, health and social service professionals	Sharing knowledge	Collaboration	Non-involvement	
Gobl et al. [66]	2019	Austria	Participatory design	Social media literacy	Serious game	71 adolescents	Sharing knowledge	Collaboration	Non-involvement	
Grant et al. [67]	2020	United Kingdom	Co-design, Public and Patient Involvement	Mental health	mHealth	26 young people	Non-involvement	Collaboration	Collaboration	
Haufe et al. [68]	2019	Netherlands	Participatory action research	Independent living	Gerontechnologies	19 seniors, 2 technology consultants and 3 municipal community builders	Collaboration	Collaboration	Collaboration	
Hernández et al. [69]	2018	Canada	Participatory design	Force Resistance Training in Hand Grasp and Arm Therapy	Video game	3 occupational therapists, 1 paediatrician, 12 children	Non-involvement	Collaboration	Non-involvement	
Hill et al. [70]	2017	Canada	Community based participatory research	Stroke education	Educational DVD	26 indigenous healthcare providers, 108 children	Collaboration	Collaboration	Collaboration	
Hiratsuka et al. [71]	2019	United States	Community-based participatory research	Posttraumatic stress disorder	Website	6 members of a native community, 5 clinical providers	Collaboration	Collaboration	Collaboration	
Hong et al. [72]	2018	United States	Co-Design	Symptom communication in chronical illness	Digital tracking tool	13 pairs of chronically ill children and their parents, 11 clinicians	Non-involvement	Collaboration	Non-involvement	
Jarke et al. [73]	2019	Germany	Co-creation	Digital neighbourhood guide	Digital neighbourhood guide	12 older adults; co-creators	Sharing knowledge	Collaboration	Sharing knowledge	
Jessen et al. [74]	2020	Norway	Participatory design	Personal strengths	mHealth app	12 patients, 15 care providers, 6 designers, 2 researchers; development team	Collaboration	Collaboration	Collaboration	
Jessen et al. [75]	2018	Norway	Co-design	Self-management of chronic conditions	mHealth self-management app	22 patients with chronic conditions	Collaboration	Collaboration	Collaboration	
Jo et al. [76]	2020	United States	Co-design	Stress management and mental well-being	mobile self-tracking platform	5 Korean mothers in the United States	Collaboration	Collaboration	Collaboration	
Jørgensen et al. [77]	2018	Denmark	Participatory design	Rheumatic diseases	e-Device for disease management	14 patients, 9 care providers	Collaboration	Collaboration	Non-involvement	
Kanstrup and Bertelsen [78]	2016	Denmark	Participatory design	Exercise technology	Exercise technology	18 young adults with depression or anxiety	Collaboration	Collaboration	Collaboration	
Kearns et al. [79]	2020	Ireland	Public patient involvement	Aphasia rehabilitation	Online user feedback questionnaire	6 patients with aphasia	Sharing knowledge	Collaboration	Collaboration	
Kenning and Treadaway [80]	2018	United Kingdom	Participatory design	Dementia	Sensory textiles	13 patients	Sharing knowledge	Collaboration	Sharing knowledge	
Kildea et al. [81]	2019	Canada	Co-design	Patient portal	smartphone app	445 stakeholders; co-leadership	Collaboration	Collaboration	Collaboration	
Kim et al. [82]	2019	Korea	Participatory design	Autism	Digital self-tracker platform	5 patients with autism and their partners	Non-involvement	Collaboration	Collaboration	
LaMonica et al. [83]	2020	Australia	Participatory design	Mental health services	Web-based platform	49 veterans	Non-involvement	Collaboration	Sharing knowledge	
Lan Hing Ting et al. [84]	2020	France	Human-centred participatory design	Fall prevention	Balance assessment tool	5 elderly people	Collaboration	Collaboration	Collaboration	
Larsen et al. [85]	2016	Denmark	Participatory design	Telemedicine	Shared service centre	35 clinicians, 29 representatives, 4 patients	Collaboration	Collaboration	Collaboration	
Latulippe et al. [86]	2020	Canada	Co-design	Social health inequalities	eHealth tool for caregivers	31 community workers, 22 health professionals, 33 caregivers; co-designers	Collaboration	Collaboration	Non-involvement	
Latulippe et al. [87]	2020	Canada	Social justice design	Social health inequalities	eHealth tool for caregivers	26 community workers, 18 health professionals, 30 caregivers	Sharing knowledge	Collaboration	Sharing knowledge	
Latulippe et al. [88]	2019	Canada	Co-design	Support for older persons	eHealth tool for caregivers	2 community workers, 2 health professionals, 3 caregivers	Sharing knowledge	Collaboration	Sharing knowledge	
Lee et al. [89]	2017	United States	User-centered design	Support for older adults	Social robots	5 older adults, 5 mental health therapists; co-designers	Sharing knowledge	Collaboration	Non-involvement	
Leeming and Thew [90]	2017	United Kingdom	Public patient involvement	Patient records	Federated platform	30 care organisations; steering group	Non-involvement	Non-involvement	Collaboration	
Leong and Johnston [91]	2016	Australia	Participatory design	Aging people	Social robots	8 older adults	Sharing knowledge	Collaboration	Non-involvement	
Leslie et al. [92]	2019	Canada	User-centred design	Health services	Mobile App to assist caregivers	25 family caregivers	Collaboration	Collaboration	Sharing knowledge	
Limaye et al. [93]	2018	Peru	Community based partcipatory research	Maternal mortality	Digital story telling	135 people from the community	Collaboration	Collaboration	Sharing knowledge	
Livingood et al. [94]	2017	United States	Community based participatory research	Obesity	Digital communication intervention	30 youths aged 15 to 19	Collaboration	Citizens in the lead	Collaboration	
Lo et al. [95]	2020	Canada	Symposium	Health informatics	Health information technology	37 participants	Non-involvement	Collaboration	Non-involvement	
Lockerbie and Maiden [96]	2020	United Kingdom	Co-design	Dementia	Digital quality of life model	10 care workers	Non-involvement	Collaboration	Sharing knowledge	
Lu et al. [97]	2017	4 EU countries	Co-creation	Healthy and active ageing	Assistive living technologies	Unknown number of multi-stakeholders	Non-involvement	Collaboration	Non-involvement	
Maertens et al. [98]	2017	United States	Community oriented approach	Human papillomavirus infection	Web-based intervention	20 Latina young adults, 27 Latino parents; community advisory committee	Collaboration	Collaboration	Sharing knowledge	
Martin et al. [99]	2020	3 EU countries	Co-design	Healthy lifestyle	mHealth apps	74 adolescents aged 13 to 16	Sharing knowledge	Sharing knowledge	Collaboration	
Matthews et al. [100]	2016	United States	Participatory design	Mental illnesses	Smartphone and web app	3 patient, 3 clinicians	Sharing knowledge	Collaboration	Non-involvement	
Mburu et al. [101]	2018	South Africa	Co-design	Preterm infants	Digital communication	17 mothers, 17 NICU nurses, 4 doctors	Sharing knowledge	Collaboration	Non-involvement	
McMillan et al. [102]	2018	United Kingdom	Public patient involvement	Health check results	website	19 members of the public; co-facilitators	Citizens in the lead	Collaboration	Sharing Knowledge	
McNaney et al. [103]	2017	United Kingdom	Co-design	Dementia	Mobile app	14 young family members of people with dementia	Collaboration	Collaboration	Sharing knowledge	
Miller et al. [104]	2016	United States	Participatory design	Health information	Information technology	8 parents, 2 children, 3 clinicians	Collaboration	Collaboration	Sharing knowledge	
Mrklas et al. [105]	2020	Canada	Co-design	Knee osteoarthritis	MHealth app	4 patients, 5 physicians, 3 trainees, 2 decision makers; research coalition	Collaboration	Collaboration	Collaboration	
Naeemabadi et al. [106]	2020	Denmark	Participatory design	Total knee replacement	Telerehabilitation program	8 patients, 4 health professionals, 4 student assistants, 2 software developers	Collaboration	Collaboration	Non-involvement	
Payton [107]	2016	United States	Co-creation	Health information HIV	Social media	15 participants	Non-involvement	Citizens in the lead	Non-involvement	
Rabba et al. [108]	2020	Australia	Participatory action research	Autism	Web-based resource	3 parents, 9 clinicians; advisory group	Collaboration	Collaboration	Non-involvement	
Rai et al. [109]	2020	United Kingdom & Ireland	Public patient involvement	High blood pressure	Digital intervention	40 patients and partners, 35 care professionals; co-investigators	Collaboration	Collaboration	Collaboration	
Rathnayake et al. [110]	2020	Australia	Co-design	Dementia	mHealth application	10 carers	Non-involvement	Collaboration	Non-involvement	
Reade et al. [111]	2017	United Kingdom	Public patient involvement	Joint pain; rheumatoid arthritis	Smartphone app	20 patients	Collaboration	Collaboration	Collaboration	
Russ et al. [112]	2020	United Kingdom	Participatory action research	Surgical care	Smartphone app	42 patients; steering group	Collaboration	Non-involvement	Collaboration	
Sin et al. [113]	2019	United Kingdom	Co-design	Psychosis	eHealth intervention	14 experts; advisory group	Non-involvement	Collaboration	Non-involvement	
Skovlund et al. [114]	2020	Denmark	Public patient involvement	Metastic melanoma	Dialogue tool	5 patients	Collaboration	Collaboration	Collaboration	
Solem et al. [115]	2020	Norway	User centred design	Chronic pain	eHealth intervention	20 project team members, 33 external stakeholders	Collaboration	Collaboration	Collaboration	
Synnot et al. [116]	2018	Australia	Participation in research	Multiple sclerosis	Website	83 patients; advisory group	Collaboration	Sharing knowledge	Collaboration	
Tsvyatkova and Storni [117]	2019	Ireland	Participatory design	Type 1 diabetes mellitus	eBook	24 parents, 20 children; design partners	Non-involvement	Collaboration	Sharing knowledge	
Tuckett et al. [118]	2018	Australia	Citizen science	Physical activity for mental health	Mobile application	8 older adults aged 65 and older; citizen scientists	Collaboration	Collaboration	Collaboration	
Vacca [119]	2019	United States	Participatory design	Teen-caregiver relationship	Digital media creation platform	8 Latina teens	Non-involvement	Citizens in the lead	Sharing knowledge	
Vallentin-Holbech et al. [120]	2020	Denmark	Co-creation	Health promotion; alcohol prevention	Virtual Reality (VR) simulation	9 students; development team	Non-involvement	Collaboration	Non-involvement	
Van Leersum et al. [34]	2020	Netherlands	User-centred design	Long-term care	Web-based preference elicitation tool	21 end-users; development team	Collaboration	Citizens in the lead	Sharing knowledge	
Vilarinho et al. [121]	2017	5 EU countries	Co-design	Cystic fibrosis	mHealth application	14 children, 15 parents, 12 health professionals	Non-involvement	Collaboration	Non-involvement	
Wikman et al. [122]	2018	Sweden	Participatory action research	Cancer	Internet-administered intervention	6 parents, 2 clinical psychologists; research partners	Non-involvement	Collaboration	Sharing knowledge	
Yarosh and Schueller [123]	2017	United States	Participatory design	Children well-being and resilience	Computing technologies	12 children; investigators	Non-involvement	Collaboration	Non-involvement	

From the 83 included records, eight were published in 2016, 14 in 2017, 15 in 2018, 17 in 2019, and 29 in 2020. Most studies were conducted in the United States (N = 17) [44, 46–48, 52, 54, 71, 72, 76, 89, 94, 98, 100, 104, 107, 119, 123], followed by Canada (N = 14) [42, 43, 45, 59, 65, 69, 70, 81, 86–88, 92, 95, 105], the United Kingdom (N = 12) [51, 56, 57, 67, 80, 90, 96, 102, 103, 111–113], Denmark (N = 6) [77, 78, 85, 106, 114, 120], and Australia (N = 6) [83, 91, 108, 110, 116, 118]. Five studies were conducted in multiple countries [53, 97, 99, 109, 121], three studies were conducted in Sweden [Erlingdottir et al. ; Gardsten; Wikman et al.], and Norway [74, 75, 115], and 2 in Austria [61, 66], the Netherlands [34, 68], and Ireland [79, 117]. One study was conducted in Belgium [49], India [50], Italy [58], Brazil [62], Spain [64], Germany [73], Korea [82], France [84], Peru [93], and South Africa [101].

One of the 83 records was based on a quantitative research design, all other included record had a research design based on qualitative methodologies. Participatory design was most prominent (N = 24) [42, 44, 45, 50, 57, 61, 63, 66, 69, 74, 75, 77, 78, 80, 82, 83, 85, 91, 100, 104, 106, 117, 119, 123], followed by co-design (N = 16) [55, 64, 65, 72, 76, 81, 86, 88, 96, 99, 101, 103, 105, 110, 113, 121]. Other often used terms were community based participatory research (N = 9) [48, 52–54, 56, 70, 71, 93, 94], co-creation (N = 8) [43, 46, 58, 62, 73, 97, 107, 120], public and patient involvement (N = 7) [67, 79, 90, 102, 109, 111, 114], participatory action research (N = 5) [59, 68, 108, 112, 122], and user centered design (N = 4) [34, 89, 92, 115]. Citizen science was only used in 2 records [49, 118], and usability study [47], co-production [51], human centered participatory design [84], social Justice design [87], symposium [95], community oriented approach [98], and participation in research [116] were all only used once to describe the research design.

Besides the term citizen, participant, patient, actor or (end) user, there are different terms used to describe the involved citizens. These include key informants [45], co-leadership [46, 81], co-designers [45, 65, 86, 89], co-researchers [51], co-creators [58, 73], co-facilitators [102], co-investigators [109], inventors [123], design/research partners [117, 122], citizen scientist [118], community advisory committee [98], research coalition [105], steering group [71, 90, 112], (expert) advisory group [51, 108, 113, 116], and development group/team [34, 59, 74, 120]. Fifty-nine of 83 records included adults, 11 records included younger adults (13 to 25 years old) [43, 51, 58, 59, 67, 78, 94, 98, 99, 119, 120], eight records included children (12 years or younger) [61, 69, 70, 72, 104, 117, 121, 123], and nine records included older adults [59, 68, 73, 83, 84, 89, 91, 97, 118]. The number of involved citizens is variable among all records, most studies involved between 30 and 40 citizens, with a highest amount of 6246 [49] and the lowest of five citizens [76, 82, 84, 114].

Research phases and citizen involvement

Full texts were read of 194 records. A total of 111 records involved citizens solely on the levels of sharing knowledge or non-involvement. In these records researchers were in the lead during all research phases and decisions. Sixty-five records were considered to have citizen involvement on a collaboration level in one or more research phases. There were 8 records analysed as having citizens in the lead in one research phases. These records had citizens who were responsible and made decisions regarding a development process or were themselves the initiators of a project. Most records involved citizens during the data collection phase. Forty of 83 records had user inclusion in all three research phases, 11 records involved citizens in the data collection and evaluation phase, ten records in the preparation and data collection phase, and one record in the preparation and evaluation phase. Thirteen records described citizen involvement only in one research phase. Table 3 shows the level of citizen involvement in the different research phases of all 194 included for full text reading. Five records are further discussed in the next section as example cases of citizen involvement in research regarding health, care or well-being.Table 3 Number of records for each level of citizen involvement and within each research phase

	Preparation of research	Data collection	Data analysis & evaluation	
Non-involvement	77	20	80	
Sharing knowledge	62	83	66	
Collaboration	39	69	33	
Citizens in the lead	1	7	0	

Example cases with high level of involvement in multiple phases

I In this section five example cases are elaborated upon to show a diversity of topics, methods, and citizen inclusion. These cases are chosen based on inclusion of citizens in multiple research phases and at a level of collaboration or higher in one or more research phases. The aim of these cases is to analyse the citizen science method during the innovation process in the domain of health, care or well-being. How were citizens involved, who were involved, what worked well, what kind of problems were met, and what kind of motivation do researchers or citizens have?

Case 1. An online self-management tool for spinal cord injury [45]

The technology in this case is an online tool to promote self-management in order to avoid rehospitalization of patients after spinal cord injury (SCI). The data was collected in 2015 and 2016 involving Canadian citizens of different provinces. This case is chosen because it shows a clear description of the organisation and collaboration between the different stakeholders. The authors included several teams demonstrating a divers and broad network (see Box 1 for a case description).Box 1. Case description:

Online self-management tool for spinal cord injury

Self-management tools are available for many chronic conditions. The use of these tools is linked to a decrease in rehospitalization rates. However, satisfaction with most self-management tools is low, and there is no known online tool for users with SCI. The goal in this study was to apply a participatory design process in order to develop an online self-management tool for users with SCI. This process was chosen by the authors to define design constraints and solutions. Sixteen individuals, seven researchers and nine persons with SCI, participated in ten different meetings of the participatory design process. Experiences with self-management were discussed in the first meetings (exploration). Thereafter, features for a new online tool were defined (discovery). These features were translated and embedded in a prototype which the group discussed in following meetings, and feedback was used to make iterations (prototyping). The design process and collaboration with potential users was valuable to engage end users in the design of a tool to promote self-management after SCI

	

In this case, the participatory design process consisted of three steps: exploration, discovery, and exploration [124]. The design process was described as an interactive design in which the knowledge of end-users was acknowledged and taken into the design by involving users. The process should include expressing tacit knowledge and encouraging sensitivity [45]. The design process included a core co-design and co-development team meeting on regular basis. The record clearly shows the connections between this core team and the council. This council consisted of a consumer advisory group of users with SCI and a product advisory group of researchers and a clinician [45]. Several other records described a collaboration between different groups, for example Wikman et al. included parent research partner group and an expert research partner group [122], Russ et al. made use of a steering group in which public representatives collaborated with researchers [112], and Mrklas et al. had a research coalition [105]. A difference with the study of Allin et al. , quite often these advisory boards did not include the actual users, but healthcare professionals [45].

The minimal involvement of citizens in the evaluation phase could be seen as a limitation of this case, but in all other phases it was very strong. Furthermore, they recognize possible bias due to a low number of involved citizens, and we noticed a lack of involvement of citizens with a lower educational level. Allin et al. discuss that all involved citizens had access to high-speed internet, which is associated with a high self-reported health status, and therefore these involved citizens are maybe less in need of self-management than those who are not online [45].

Beforehand the authors motivated their choice to use a participatory design, because it might empower and educate the citizens, and encourages them to adopt the developed application. The three benefits reported afterwards include “elicitation and consideration of diverse accessibility considerations”, “prioritization of features and identification of core design concerns”, and “co-creation of acceptable strategies and techniques to mitigate identified concerns” [45]. Although the expected benefits considered the citizens, the experienced benefits show the value for the development process.

Case 2. Digital stories for health promotion [59]

In this case, digital stories are the technologies, which are at the same time the technologies under development as well as the technologies used to perform the research. Sharing knowledge on healthy lifestyle and increasing self-esteem are the health goals with the stories. From 2012 to 2014, 18 workshops on digital stories were organised in communities from Victoria to Ahousat in Canada. This second case was chosen due to intensive groupwork and collaboration between younger and older adults within the community as researchers. A mutual learning process was present between the younger and older adults to develop the digital stories (see Box 2 for a case description).Box 2. Case description:

Digital stories for health promotion

Aboriginal youth in Canada experience lower health and well-being due to different challenges they face. Digital stories can emphasize on community connection and cultural continuity, which are factors influential to improve health and well-being. The aim of the study was to create and share digital stories to support health promotion and encourage interactions within a community. This creates a place for younger and older adults where knowledge on healthy lifestyle could be shared, where mentorship could be facilitated, and where self-esteem could be increased by celebrating identity and community. Participatory action research was used to engage younger and older adults in the community. A youth research team was formed with eight younger adults. Next to this team there were 60 core younger participants, 170 younger workshop participants, and 14 older adults involved. At the start, the youth research team was trained to create digital stories. After workshops in which the older adults taught about their healthy lifestyle, the younger and older adults together created mini stories with the use of cameras, audio recorders, and story boards. These stories were evaluated during follow-up sessions, and a small group of younger adults were trained to lead other communities in creating their digital stories

	

Fletcher and Mullett [2016] refer to McIntyre, but do not give a definition for their research method [59, 125]. The authors argue that participatory action research was beneficial because it was a non-prescriptive approach. Other records using similar participatory action research methods do give a definition. For example Wikman et al. based their design on Kindon et al., “which is a collaborative process of knowledge production and co-learning, placing people with lived experience at the centre of the process” [122, 126], and Rabba et al. [2020] based their method on Baum et al., “the stakeholders are integral to all parts of the research process, participants’ active role in the outcome sets participatory action research apart from traditional research methods where the participants tend to be more passive in their receipt of research outcomes” [108, 127]. Often a participatory action research is divided in phases. Rabba et al. used planning, action, observation, reflection, and using the new learning to plan further steps in their research [108], and Haufe et al. used the two phases of understanding the current and developing a tool [68]. All participants of the study by Fletcher and Mullet were involved in the phases planning, designing, and evaluation of the community workshops [59].

The research process gave the community members together with the researchers the opportunity to create a unique, broad, and holistic concept of healthy lifestyle, including personal wellness in body, mind, and spirit. Clear outcome and follow-up include ownership of the stories by the communities, the youth took on leadership in training other communities to develop digital stories, and the older adults took a teaching role in ecology, traditional food preparation, cleansing rituals, and medicinal plants [59]. There were more records discussing the need for public health interventions and the involvement of community members in creating digital stories based on their community and personal stories. Although in most records the creation was done by the community members, this creation was more seen as a needs assessment activity and the researchers chose which stories were used in discussions [93].

It seems that the aim to use a participatory action approach with digital stories to improve health and well-being was positive. The creation of stories by community members was relevant to capture a personal voice and made explicit which public health themes were recognized. The workshops were fun and engaging, and the youth experienced a process with the older adults giving them a voice and a sense of belonging. However, Fletcher and Mullett also acknowledge that the participants were very active in the creation of stories, presenting, training, and workshops, but there was a much lower interest in planning and analysis [59].

Case 3. Design social play things [61]

This case is about technology for social play for mixed groups of neurodivergent and neurotypical children. The well-being goal is to scaffold and support the development of complex social skills through technology-facilitated social play. In the ‘SocialPlayTechnologies’ project the researchers collaborated over the course of three years with two inclusive mainstream primary schools in Vienna, Austria. Three social play technology prototypes were designed together with three groups of children, aged 7 to 12 years. We selected this case due to researchers' effort to give children the lead in the collaborative design process, their flexibility in adapting to the children’s needs, and to showcase how citizen science can be executed with challenging target groups (see Box 3 for a case description).Box 3. Case description:

Design social play things

Social play is an essential part of children’s healthy development. For neurodivergent children engaging in social play with neurotypical peers that have different interaction styles can be challenging, often leading to a preference for solitary play. This study sought to explore how digital technologies can facilitate and support social play among mixed groups of neurotypical and neurodiverse children. The aim was to create technologies that are engaging for both target groups, interactive, open-ended, robust, embedded in their natural play context, extendable, and non-normative. The researchers engaged in a long-term participatory design (PD) process with three groups of children aged 7 to 12 years in. In total 16 children participated as co-designers in this study. The PD process consisted of weekly or bi-weekly series of design workshops (50 in total across the three groups) with at least two research team members as facilitators. Groups were completely free in what social play technology they would design. The first phase of the PD process focussed on exploring children’s perception of playfulness. This phase was followed by various technology immersion activities to introduce the children to sensors and actuators. In the next phase, researchers provided new input through material, scenarios, and narratives, while also narrowing the design space to agreed-upon ideas. When concepts were more concrete, researchers started to bring in first prototypes or groups collaboratively designed prototypes. Three distinct social play technologies were designed: 1) a concept with coloured fabrics embedded with interactive lighting called LightSpaces, 2) a set of interactive pads that trigger sounds called MusicPads, and 3) a concept consisting of a reading lamp with a Raspberry Pi, a camera and a projector called PictureStage. Towards the end of the design process researchers worked on increasing the maturity of the prototypes for the purpose of evaluation. This evaluation phase is escribed in another paper

	

The researchers applied a PD to reach their design goal of creating social play technologies suitable for the different ways neurodivergent and neurotypical children make sense of the world, can only be achieved by including these children directly in the design process [61]. As neurotypical adult researchers they felt unequipped to imagine the children’s distinct sense making process in social play. Similar reasoning on the value of PD is given by [42] who argue that PD ensures that a design solution meets the target groups needs and beliefs, and is culturally relevant. Besides understanding perspective and producing relevant design solutions, PD can provide other benefits to participants, such as fostering agency and teaching useful skills [123]. The researchers’ design process was built on previous PD work with children including co-operative Inquiry [128]. Co-operative inquiry has three key elements: (1) children are regarded as research partners and involved throughout the design process, (2) design activities with low-tech materials, and (3) technology immersion in which children explore the design space of novel technologies. Yarosh and Schueller also worked with children and showed methodological similarities by having co-operative inquiry as a framework to build their PD process, and having long-term engagement since they met with the children twice a week for 14 design sessions [123].

Another concept, central in the researchers’ design process, is ‘Handlungsspielraum’ (in English: room to act) [129]. To open up Handlungsspielraum it was important for the researchers to create design activities that balance structure and freedom through the used materials, planned activities, physical setting and role of the facilitator. While structure provides safety, guides participants, and offers stimuli to participant, freedom allows participants to explore their creativity and ideas. Researchers found that this balance between structure and freedom had a similar effect on creativity within play. Hence, this balance was also reflected in the final prototypes through modularity, manipulability and the option to use the prototypes without the technological component. The researchers in this case tried to let children lead the design process. This is reflected in the fact that each group came up with a different social play technology. Moreover, while the design process followed the same overall blueprint across design groups, the overall pace and specific content of each design session was different for each group and evolved based on the children’s interest and conceptions of playfulness. On the other hand, the researchers noted that they sometimes made design decision contrary to the wishes of the children to limit the complexity of the designed technologies.

Researchers in the current case indicated that working with a mixed group of neurotypical and neurodivergent children was challenging, especially in the groups where no teacher was present. Researchers had to act as mediators when conflict occurred between children and prepared for this task by drawing on previous literature about appropriate design frameworks for this target group. Moreover, they interviewed the teachers at the start of the design process to learn about the group dynamics. Some team members received a training from a special needs educator to learn how to deal with conflict.

Case 4. New voices to design exercise technology [78]

The technology in this case is a health promoting digital application to support vulnerable young adults in exercising and physical activities. The ‘Pulse Up project’ is a collaboration of a general practitioner (GP), community workers, and citizens in a neighbourhood in Denmark. Eighteen young adults, aged 18–30, suffering from depression and/or anxiety, and having a low level of physical activity, participated in the exploration and manifestation of design ideas. This case is chosen because of the situated PD process, the creative PD activities, and the inclusion of vulnerable citizens in the choice of research activities, design ideas, analysis, and evaluation of each step in the process (see Box 4 for a case description).Box 4, case description

New voices to design exercise technology

Most research into digital technologies to support exercise is centred on people already engaged in exercise activities, while those who are challenged to exercise and are at risk of ill health due to low physical activity level have a limited voice in technology design. This research presents a PD process with young adults suffering from depression and/or anxiety, living in a neighbourhood identified as a high-risk health zone, and motivated but faced with challenges to exercise. The design process consisted of two interventions. The first intervention focussed on developing appropriate participatory methods for bringing new voices to the design of exercise technology and to gain initial insight to inform future design. The second intervention focussed on exploring and manifesting design ideas. Participants met twice a week for eight weeks in the neighbourhood’s local gym to exercise with an instructor and participate in a total of 16 PD activities. Eleven young adults participated in the first intervention. Four continued in the second intervention, and seven new participants were recruited. Exercise itself was not challenging, but the challenge was in the sub-activities to complete before exercise. The participants created three visions to meet with this challenge: 1) dragging friends to exercise, 2) keeping an eye on the activities of group members, and 3) a ‘join’ functionality as an easy way to join activities. The participants’ visions on exercise technology were low since they did not prioritize tracking and competition, health information or exercise instructions. However, based on their challenge, they created visions for digital support

	

To bring the voices of vulnerable young adults into the design process of digital technologies to support health promotion, Kanstrup et al. chose for a situated PD [78]. The process combined physical exercises with PD activities. Although the researchers initially planned to do the PD activities in a classroom setting, the participants expressed a dislike of classrooms and preferred activities in the gym. Therefore, the PD activities were integrated with the physical activities in the gym. To change the setting, they made rules for the activities to emphasise on the need for professionalism, effectiveness, and the ability to trigger design. This formed the foundation for the development of a design cycle with progressive stages, supported by game-like PD activities, continuous analysis and evaluation of results. All activities were developed as a fusion of training sessions with music and exercises, and PD activities. In addition, Kanstrup et al. developed a variation of artefacts to trigger exercise and technology design [78].

The previous case had a similar approach in their research of social playthings. Both the location and the integration of playful activities became part of the design process. Yarosh and Schueller also reported that they deviated from their original plan of putting the researchers in charge of video-recording design sessions [123]. They decided to give the participants control over the recording process to change the perceived power-imbalance between researchers (as data-collector) and participants (as data-objects). These studies show the benefits of situated research in which group activities are part of the research process. Although this situated approach can be in a physical place, it can also be interpreted as a safe space for open discussions, as is visible in the study of Dewa et al. [2020]. In their study with young people with mental health, a WhatsApp-group became the place for open discussions between the co-researchers [51].

Each PD intervention was evaluated by the participants, who were engaged in an iterative analytical process. Visualisations of analytical results were shared with the participants between 2 interventions, and they were invited to reflect on the presented analysis and to reconstruct any misinterpretations, missed insight or future visions. After the second intervention, the co-researchers conducted a final analysis of the visions, in which they identified design elements for exercise technology. The created visions move beyond an understanding of exercise technology for vulnerable groups as simply information apps, towards an ambition to design mediators of community health resources in which digital communication is the mediator for physical activities. This requires critical insight about adolescent use of digital communication and the potential importance of messaging, apps, gaming, wearable technology, and rapid changes in youth communication and use of digital technology in developing adolescent physical activity health promotion. This is visible in the study of Livingood et al. in which they develop a digital communication intervention to reduce adolescent obesity [94].

Case 5. A person-centred patient portal [81]

This study focuses on a patient portal app for patients with cancer. A patient portal is a secure extension of an electronic medical record (EMR) of a health care institution where patients and professionals have access to [130]. In the study of Kildea et al. participatory stakeholder co-design was applied, involving patients and health care providers [81]. Data were collected between 2015 and 2018. This study was selected because of the interface between patients and the medical world, and the authors strongly show their used participatory stakeholder co-design (see Box 5 for a case description).Box 5. Case description

Design and development of a patient portal (eHealth)

Patient portals represent a real-world example of patients facing electronic health (eHealth). Patient portals become important in daily care because they provide patients with personal health information and contribute to patient engagement and empowerment. A participatory stakeholder co-design was applied involving patients and health care providers. Six core elements were used: equal co-leadership, patient preference determination, security, governance, and legal input, user evaluation and feedback, continuous staff input, and end-user testing. Regarding ‘equal co-leadership’, all three co-leads, a patient, a radiation oncologist, and a medical physicist, were equally involved in decisions and in constant communication. ‘Patient preference determination’ was conducted by a voluntary convenience sampling survey, to obtain input from people receiving cancer treatment. For ‘security, governance and legal input’ purposes a Security and Governance team provided guidance regarding the security and confidentiality of patient data and compliance with applicable regulations. Three patients were participated in a focus group to provide ‘user evaluation and feedback’. The prototype was demonstrated, and participants were observed. A second focus group was conducted before pilot release, consisting of five members of the cancer centre’s patients committee. ‘Continuous staff’ input was ensured by a clinical co-lead, radiation therapists, staff ranging from health care providers to senior management, medical physicists, oncology nurses, radiation oncologists, administrative assistants, and the board of directors of the institution. Finally, to ensure ‘end-user testing’, students and real patients tested the first version of the prototype

	

In this case the differences between co-design, patient-centred design and person-centred design were described. In co-design, patients help identify the process or project that needs to be designed based on personal experience. Patient-centered design focuses on fulfilling the needs of the patients, but the project may not have been identified by patients or involve patients. Person-centered design focuses on patients’ need as a whole person and as an equal partner in their care [131, 132]. Kildea et al. described that the term patient co-design is often confused with patient-centered design and person-centered design [81]. Therefore, the term participatory stakeholder co-design was preferred as term in this study. Patients and clinicians were involved as equal partners, and both actively participate in all parts of the design process [81]. Some other studies involving both patients and health care providers used the term PD [42, 57, 74, 77, 85, 100, 104, 106, 108].

Furthermore, in this case an agile development approach was applied, which consisted of rapid prototyping and testing of various features [133]. A team of developers, students in computer science and medical physics, and co-leads worked together. This led towards a person-centered, clinician acceptable, and informatics feasible patient portal. Comparable to this case, in the study of Leeming and Thew a patient data platform was developed involving different stakeholders as part of a steering group. Public and patient views were explored during design sessions, and the steering group highlighted the key learnings from the feedback to guide the design [90].

Kildea et al. described several challenges during the process of this study [81]. For example, the equal co-leadership, where patients are on an equal level as clinicians. This was often not clear to people outside of the immediate design and development team because the hierarchical health care system assumed that projects must be led by a clinician. Also, challenges regarding security, governance, and legal input, especially time-consuming legal issues, were encountered, for example intellectual property, liability, and the contents of the patient disclaimer form. However, as a result of this study a final design of the patient portal smartphone app for patients with cancer was developed from scratch within the health care system based on the principles of person-centeredness design.

Discussion

Main findings

The aim was to provide insight in (1) the levels of citizen involvement in current research on technological innovations for health, care or well-being, (2) the used participatory methodologies, and (3) lesson’s learned by the researchers as described in identified records. From the 194 records included after full text reading, 83 records involved citizens on a collaboration level in at least one research phase. The level of citizen involvement differed across research phases. In contrast to the review of Domecq et al. in which citizen involvement was lacking or tokenism in the data collection phase, in our scoping review most citizens were involved in the data collection, and less in preparation or evaluation [29].

The metrics of these 84 records show that there was a two-fold increase in one year between 2019 and 2020. This shows a growing interest in citizen science for health, care or well-being in the last years. Health, care or well-being research was originally considered less adequate for citizen science, but exceptions are arising [13]. The increase is also visible in the scientific journals that are picking up citizen science as a separate field. The journal ‘Citizen Science Theory and Practice’ was a first journal on citizen science, with a first issue in 2016 [134]. Other journals, such as PlosONE, include citizen science as a search or subject area. In the domain of health, care or well-being the use of citizen science was also incorporated recently, considering the entry of the Medical Subject Heading (MeSH) in 2020. Looking at the countries where the citizen science studies were performed, most of the 83 records in our sample originated from developed countries. In the entire dataset (N = 3846), there is a lot of community-based research in developing countries, and in most research the citizens are involved, but most records were excluded due to lack of technological innovation [135–138]. The included studies show a variation in the number of citizens involved, but most studies did not include more than 20. Although the number of citizens in the studies is often argued as sufficient or data saturation was reached, it introduces the risk of recruiting only a minority of the population, often the well-educated and most affluent citizens [37].

Apparent from all records was the terminology used to describe the research methodology or the citizen involvement. The term citizen science was only used in two records and participatory design was most prominent in the included records. Identical authors give different terms to their research design across studies [42, 43, 46, 47, 86, 87], and when using similar terms, the authors base the research method on different sources [86, 88]. Inclusion of all the different research methods is due to our broad search string, however, this is in line with continuing discussions on the definition for citizen science [11], and the acceptance of working in citizen science with different terms and methodologies [12]. Den Broeder et al. also wrote “some approaches in health research strongly resemble citizen science, one of these is participatory action research” [5]. However, they argue that there is a main difference between participatory action research and citizen science, and that difference is in the action part, which refers to the need to act, having change, addressing a specific problem, or developing an intervention. Citizen science can also be a method used to do research without a focus on any action. Furthermore, participatory action research aims towards a strong involvement of citizens and in citizen science, the citizens can be engaged in a less intensive way [5]. Regarding the two records naming their research method citizen science, the level of involvement was diverse. One aimed to reach a large audience with a citizen science-mass experiment with no focus on action [49], and the other trained citizen scientists to become agents of change in their own environment which had a focus on action [118]. These examples show that there is no consensus on the use and terminology of citizen science and related methodologies.

These different views and approaches towards citizen science raise questions regarding the value and benefits of citizen science. Greenwood et al. argue that active involvement of citizens in the preparation phase ensures that the most relevant research questions will be asked [139]. Also, successful involvement of citizens causes sensitivity and understanding of needs to incorporate in design and implementation [61, 140]. Thereby increasing the chance of adoption and sustainable implementation. Many records in this scoping review consider a form of citizen science to raise awareness on health, care or well-being issues. This is in line with the benefits of citizen science for the citizen scientists [8, 9]. However, little evaluation was done in the included empirical studies. Some discussed the possible benefits for the citizen scientists, such as improved personal health. However, these issues were not obtained from the citizens, but expected or experienced by the researchers. One record discussed that the development team had the feeling of being heard and all ideas were considered in the development [34], but it is often unknown how citizens have experienced their contributions and if they would recognize their ideas as part of a prototype [141].

Compared to traditional science, citizen science asks for different research approaches and skills from researchers. The selected cases imply that successful citizen science project develope a structural and longtidutinal partnership with their collaborators [45, 59, 61, 78, 81]. Citizen science requires an open attitude, flexibility and context sensitivity from researchers. They have to give-up traditional power dynamics, step away from their own expectations and pre-conceptions, be sensitive to the signals of their co-researcher and be willing to engage in a mutual learning experience. For example, researchers might have preconceived theoretical knowledge about the topic of interest and probably about the research population, but this does not always align with the views of the co-researchers. Yarosh and Schueller and Gobl et al. both underline that it is important that the researchers learned how to engage and learn from the mental models and views of their co-researchers [66, 123]. Being sensitive to the context and the co-researcher often means that pre-planned methods and approaches have to be adapted on the fly to create a comfortable and stimulating environment for citizens to actively engage in research [61, 78].

Conclusion

In this scoping review, our main interest was (1) the levels of citizen involvement in current research on technological innovations for health, care or well-being, (2) the used participatory methodologies, and (3) lesson’s learned by the researchers. The increasing number of publications per year in the period 2016–2020, shows the growing interest in citizens science in the field of health, care or well-being. The records that surfaced in this review confirm the diversity of research methods that are applied in citizen science for health care or well-being, mainly applying qualitative methods, often tailored to the specific research problem and group of co-researchers. The five example cases showcase how the contextuality of citizen science as a situated research approach has an impact on the chosen research methods, analysis, and research results. Citizen science in this context requires an open attitude towards how citizens are involved, who the involved citizens are, which methods are used and what kind of topics are researched.

Based on this scoping review, the topic of the required flexibility of researchers in citizen science projects is an opportunity for further research. Although citizen science projects could take more time, it has a meaningful impact on researchers and citizen scientists [Dewa et al. 2020]. Researchers need to engage with co-researchers and find a balance between control and an open structure. The records in this review show some directions to adapt and relinquish control based on learning experience during the citizen science project. Explore this learning process in more depth could provide further understanding and recommendations for researchers as well as citizen scientists involved in citizen science projects. Where is guidance of researchers asked, where is it necessary to give citizens more space, and when would you exert more supervision?

Another follow-up could include exploration of collaboration with vulnerable target groups. The cases and other records in this review show that it is possible to actively involve and collaborate with vulnerable target populations, with or without the use of proxies. Furthermore, based on the minimal findings considering the value of citizen science for the involved citizens, motivation of researchers to start collaborating and motivation of citizens to become active in a project, this could be an interesting direction to explore. Connected to this question on the value of citizen science is the question on the need of a high level of citizen involvement. This scoping review showed that the involvement of citizens in citizen science projects could differ in each research phase. From the included records, it cannot be discussed whether this difference in level of involvement effects the project, research outcome or contribution of the citizens. Despite the lower levels of involvement in one research phase or another, the contribution of citizens in other phases was worthwhile. Therefore, it might be argued that a high level of involvement is not always necessary to execute a valuable citizen science project, but we cannot draw firm conclusions without empirical evidence.

Abbreviations

eHealth Electronic health

EMR Electronic medical record

GP General practitioner

MeSH Medical Subject Heading

PD Participatory design

SCI Spinal cord injury

Acknowledgements

The authors thank all colleagues of TOPFIT Citizenlab for their regular support and collaboration. Special thanks to R. Wolkorte and L. Heesink for their involvement at the start of this scoping review and K.E. Konrad and M.E.M. den Ouden for their supervision.

Author contributions

All authors designed the study protocol. JvdZ contributed to the data collection. All authors contributed to the data analysis and data interpretation. CvL undertook the drafting of the first manuscript. All authors contributed and critically reviewed the manuscript, and all approved with the final version.

Funding

This study was powered by the Twente Regional Deal and received financial support from the central government’s regional budget, the province of Overijssel, the region of Twente, and the Twente Board. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Availability of data and materials

The datasets supporting the findings and conclusions are available from the corresponding actor on request.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Liu JX Goryakin Y Maeda A Bruckner T Scheffler R Global health workforce labor market projections for 2030 Hum Res Health 2017 15 1 11 10.1186/s12960-017-0187-2
Liu JX, Goryakin Y, Maeda A, Bruckner T, Scheffler R. Global health workforce labor market projections for 2030. Hum Res Health. 2017;15(1):11. 10.1186/s12960-017-0187-2.10.1186/s12960-017-0187-2
2. Krick T Huter K Domhoff D Schmidt A Rothgang H Wolf-Ostermann K Digital technology and nursing care: a scoping review on acceptance, effectiveness and efficiency studies of informal and formal care technologies BMC Health Serv Res 2019 19 1 400 10.1186/s12913-019-4238-3 31221133
Krick T, Huter K, Domhoff D, Schmidt A, Rothgang H, Wolf-Ostermann K. Digital technology and nursing care: a scoping review on acceptance, effectiveness and efficiency studies of informal and formal care technologies. BMC Health Serv Res. 2019;19(1):400.31221133 10.1186/s12913-019-4238-3
3. Wiggins A Wilbanks J The rise of citizen science in health and biomedical research Am J Bioeth 2019 19 8 3 14 10.1080/15265161.2019.1619859 31339831
Wiggins A, Wilbanks J. The rise of citizen science in health and biomedical research. Am J Bioeth. 2019;19(8):3–14. 10.1080/15265161.2019.1619859.31339831 10.1080/15265161.2019.1619859
4. Rosner H Data on wings Sci Am 2013 308 2 68 73 10.1038/scientificamerican0213-68 23367787
Rosner H. Data on wings. Sci Am. 2013;308(2):68–73. 10.1038/scientificamerican0213-68.23367787 10.1038/scientificamerican0213-68
5. Den Broeder L Devilee J Van Oers H Schuit AJ Wagemakers A Citizen Science for public health Health Promot Int 2018 33 3 505 514 10.1093/heapro/daw086.PubMed 28011657
Den Broeder L, Devilee J, Van Oers H, Schuit AJ, Wagemakers A. Citizen Science for public health. Health Promot Int. 2018;33(3):505–14. 10.1093/heapro/daw086.PubMed.28011657 10.1093/heapro/daw086.PubMed
6. Irwin A Citizen science: a study of people, expertise, and sustainable development 1995 New York Routledge
Irwin A. Citizen science: a study of people, expertise, and sustainable development. New York: Routledge; 1995.
7. Ceccaroni L Woods SM Sprinks J Wilson S Faustman EM Bonn A Citizen science, health, and environmental justice 2021 Cham Springer
Ceccaroni L, Woods SM, Sprinks J, Wilson S, Faustman EM, Bonn A, et al. Citizen science, health, and environmental justice. Cham: Springer; 2021.
8. Haywood B A “sense of place” in public participation in scientific research Sci Educ 2013 98 64 83
Haywood B. A “sense of place” in public participation in scientific research. Sci Educ. 2013;98:64–83.
9. King AC Winter SJ Sheats JL Rosas LG Buman MP Salvo D Leveraging citizen science and information technology for population physical activity promotion Transl J Am Coll Sports Med 2016 1 4 30 44 27525309
King AC, Winter SJ, Sheats JL, Rosas LG, Buman MP, Salvo D, et al. Leveraging citizen science and information technology for population physical activity promotion. Transl J Am Coll Sports Med. 2016;1(4):30–44.27525309
10. Auerbach J Barthelmess EL Cavalier D Cooper CB Fenyk H Haklay M The problem with delineating narrow criteria for citizen science Proc Natl Acad Sci USA 2019 116 31 15336 15337 10.1073/pnas.1909278116 31363074
Auerbach J, Barthelmess EL, Cavalier D, Cooper CB, Fenyk H, Haklay M, et al. The problem with delineating narrow criteria for citizen science. Proc Natl Acad Sci USA. 2019;116(31):15336–7. 10.1073/pnas.1909278116.31363074 10.1073/pnas.1909278116
11. Heigl F Kieslinger B Paul KT Uhlik J Dorler D Opinion: Toward an international definition of citizen science Proc Natl Acad Sci USA 2019 116 17 8089 8092 10.1073/pnas.1903393116 31015357
Heigl F, Kieslinger B, Paul KT, Uhlik J, Dorler D. Opinion: Toward an international definition of citizen science. Proc Natl Acad Sci USA. 2019;116(17):8089–92. 10.1073/pnas.1903393116.31015357 10.1073/pnas.1903393116
12. Eitzel MV Cappadonna JL Santos-Lang C Duerr RE Virapongse A West SE Citizen science terminology matters: exploring key terms Citiz Sci 2017 2 1 20
Eitzel MV, Cappadonna JL, Santos-Lang C, Duerr RE, Virapongse A, West SE, et al. Citizen science terminology matters: exploring key terms. Citiz Sci. 2017;2:1–20.
13. Kullenberg C Kasperowski D What is citizen science?–a scientometric meta-analysis PLoS One. 2016 11 1 e0147152 10.1371/journal.pone.0147152 26766577
Kullenberg C, Kasperowski D. What is citizen science?–a scientometric meta-analysis. PLoS One. 2016;11(1):e0147152. 10.1371/journal.pone.0147152.26766577 10.1371/journal.pone.0147152
14. Huang J Hmelo-Silver CE Jordan R Gray S Frensley T Newman G Scientific discourse of citizen scientists: models as a boundary object for collaborative problem solving Comput Hum Behav 2018 87 480 492 10.1016/j.chb.2018.04.004
Huang J, Hmelo-Silver CE, Jordan R, Gray S, Frensley T, Newman G, et al. Scientific discourse of citizen scientists: models as a boundary object for collaborative problem solving. Comput Hum Behav. 2018;87:480–92. 10.1016/j.chb.2018.04.004.10.1016/j.chb.2018.04.004
15. Rafner J, Gajdacz M, Kragh G, Hjorth A, Gander A, Palfi B, et al. Revisiting Citizen Science Through the Lens of Hybrid Intelligence. arXiv. 2021:34. abs/2104.14961.
16. Danielsen F Burgess ND Coronado I Enghoff M Holt S Jensen PM Hecker S Haklay M Bowser A Makuch Z Vogel J The value of indigenous and local knowledge as citizen science Citizen science: innovation in open science, society and policy 2018 London UCL Press 110 123
Danielsen F, Burgess ND, Coronado I, Enghoff M, Holt S, Jensen PM, et al. The value of indigenous and local knowledge as citizen science. In: Hecker S, Haklay M, Bowser A, Makuch Z, Vogel J, editors., et al., Citizen science: innovation in open science, society and policy. London: UCL Press; 2018. p. 110–23.
17. Koepnick B Flatten J Husain T Ford A Silva DA Bick MJ De novo protein design by citizen scientists Nature 2019 570 7761 390 10.1038/s41586-019-1274-4 31168091
Koepnick B, Flatten J, Husain T, Ford A, Silva DA, Bick MJ, et al. De novo protein design by citizen scientists. Nature. 2019;570(7761):390. 10.1038/s41586-019-1274-4.31168091 10.1038/s41586-019-1274-4
18. Snik F Rietjens JHH Apituley A Volten H Mijling B Di Noia A Mapping atmospheric aerosols with a citizen science network of smartphone spectropolarimeters Geophys Res Lett 2014 41 20 7351 7358 10.1002/2014gl061462
Snik F, Rietjens JHH, Apituley A, Volten H, Mijling B, Di Noia A, et al. Mapping atmospheric aerosols with a citizen science network of smartphone spectropolarimeters. Geophys Res Lett. 2014;41(20):7351–8. 10.1002/2014gl061462.10.1002/2014gl061462
19. Wright J Parry J Mathers J Participation in health impact assessment: objectives, methods and core values Bull World Health Organ 2005 83 1 58 63 15682250
Wright J, Parry J, Mathers J. Participation in health impact assessment: objectives, methods and core values. Bull World Health Organ. 2005;83(1):58–63.15682250
20. Burton P Conceptual, theoretical and practical issues in measuring the benefits of public participation Evaluation 2009 15 3 263 284 10.1177/1356389009105881
Burton P. Conceptual, theoretical and practical issues in measuring the benefits of public participation. Evaluation. 2009;15(3):263–84. 10.1177/1356389009105881.10.1177/1356389009105881
21. Ianniello M Iacuzzi S Fedele P Brusati L Obstacles and solutions on the ladder of citizen participation: a systematic review Public Manag Rev 2019 21 1 21 46 10.1080/14719037.2018.1438499
Ianniello M, Iacuzzi S, Fedele P, Brusati L. Obstacles and solutions on the ladder of citizen participation: a systematic review. Public Manag Rev. 2019;21(1):21–46. 10.1080/14719037.2018.1438499.10.1080/14719037.2018.1438499
22. Carpini MXD Cook FL Jacobs LR Public deliberation, discursive participation, and citizen engagement: a review of the empirical literature Annu Rev Polit Sci 2004 7 315 344 10.1146/annurev.polisci.7.121003.091630
Carpini MXD, Cook FL, Jacobs LR. Public deliberation, discursive participation, and citizen engagement: a review of the empirical literature. Annu Rev Polit Sci. 2004;7:315–44. 10.1146/annurev.polisci.7.121003.091630.10.1146/annurev.polisci.7.121003.091630
23. Conrad CC Hilchey KG A review of citizen science and community-based environmental monitoring: issues and opportunities Environ Monit Assess 2011 176 1–4 273 91 10.1007/s10661-010-1582-5 20640506
Conrad CC, Hilchey KG. A review of citizen science and community-based environmental monitoring: issues and opportunities. Environ Monit Assess. 2011;176(1–4):273–91. 10.1007/s10661-010-1582-5.20640506 10.1007/s10661-010-1582-5
24. Garau C Citizen participation in public planning: a literature review Int J Sci 2012 1 12 21 44
Garau C. Citizen participation in public planning: a literature review. Int J Sci. 2012;1(12):21–44.
25. Irvin RA Stansbury J Citizen participation in decision making: is it worth the effort? Public Admin Rev 2004 64 1 55 65 10.1111/j.1540-6210.2004.00346.x
Irvin RA, Stansbury J. Citizen participation in decision making: is it worth the effort? Public Admin Rev. 2004;64(1):55–65. 10.1111/j.1540-6210.2004.00346.x.10.1111/j.1540-6210.2004.00346.x
26. Kimura AH Kinchy A Citizen science: probing the virtues and contexts of participatory research Engaging Sci Technol Soc 2016 2 331 61 10.17351/ests2016.99
Kimura AH, Kinchy A. Citizen science: probing the virtues and contexts of participatory research. Engaging Sci Technol Soc. 2016;2:331–61.10.17351/ests2016.99
27. McGuire M Collaborative public management: assessing what we know and how we know it Public Admin Rev 2006 66 S1 33 43 10.1111/j.1540-6210.2006.00664.x
McGuire M. Collaborative public management: assessing what we know and how we know it. Public Admin Rev. 2006;66(S1):33–43. 10.1111/j.1540-6210.2006.00664.x.10.1111/j.1540-6210.2006.00664.x
28. Peter M Diekotter T Kremer K Participant outcomes of biodiversity citizen science projects: a systematic literature review Sustainability-Basel 2019 10.3390/su11102780
Peter M, Diekotter T, Kremer K. Participant outcomes of biodiversity citizen science projects: a systematic literature review. Sustainability-Basel. 2019. 10.3390/su11102780.10.3390/su11102780
29. Domecq JP Prutsky G Elraiyah T Wang Z Nabhan M Shippee N Patient engagement in research: a systematic review Bmc Health Serv Res 2014 10.1186/1472-6963-14-89 24568690
Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, et al. Patient engagement in research: a systematic review. Bmc Health Serv Res. 2014. 10.1186/1472-6963-14-89.24568690 10.1186/1472-6963-14-89
30. Malterud K Elvbakken KT Patients participating as co-researchers in health research: a systematic review of outcomes and experiences Scand J Public Health 2020 48 6 617 628 10.1177/1403494819863514 31319762
Malterud K, Elvbakken KT. Patients participating as co-researchers in health research: a systematic review of outcomes and experiences. Scand J Public Health. 2020;48(6):617–28. 10.1177/1403494819863514.31319762 10.1177/1403494819863514
31. Arnstein SR Ladder of citizen participation J Am I Planners 1969 35 4 216 224 10.1080/01944366908977225
Arnstein SR. Ladder of citizen participation. J Am I Planners. 1969;35(4):216–24. 10.1080/01944366908977225.10.1080/01944366908977225
32. Haklay M Citizen science and volunteered geographic information: overview and typology of participation 2013 Dordrecht Springer
Haklay M. Citizen science and volunteered geographic information: overview and typology of participation. Dordrecht: Springer; 2013.
33. Kotus J Position of the Polish city on the ladder of public participation: are we going the right way? The case of Poznan Cities 2013 35 226 236 10.1016/j.cities.2013.08.001
Kotus J. Position of the Polish city on the ladder of public participation: are we going the right way? The case of Poznan. Cities. 2013;35:226–36. 10.1016/j.cities.2013.08.001.10.1016/j.cities.2013.08.001
34. van Leersum CM Moser A van Steenkiste B Reinartz M Stoffers E Wolf J What matters to me—a web-based preference elicitation tool for clients in long-term care: a user-centred design BMC Med Inform Decis Mak 2020 20 1 57 10.1186/s12911-020-1067-6 32183786
van Leersum CM, Moser A, van Steenkiste B, Reinartz M, Stoffers E, Wolf J, et al. What matters to me—a web-based preference elicitation tool for clients in long-term care: a user-centred design. BMC Med Inform Decis Mak. 2020;20(1):57. 10.1186/s12911-020-1067-6.32183786 10.1186/s12911-020-1067-6
35. Connor DM A new ladder of citizen participation Natl Civ Rev 1988 77 3 249 257 10.1002/ncr.4100770309
Connor DM. A new ladder of citizen participation. Natl Civ Rev. 1988;77(3):249–57. 10.1002/ncr.4100770309.10.1002/ncr.4100770309
36. Heyen NB Self-tracking as knowledge production: quantified self between prosumption and citizen science 2016 Wiesbaden Springer VS
Heyen NB. Self-tracking as knowledge production: quantified self between prosumption and citizen science. Wiesbaden: Springer VS; 2016.
37. Ferro E, Molinari F. Making Sense of Gov 2.0 Strategies: ’No Citizens, No Party’. Prosser A, Parycek P, editors. Vienna. 2009. 10.29379/jedem.v2i1.13
38. Munn Z Peters MDJ Stern C Tufanaru C McArthur A Aromataris E Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach Bmc Med Res Methodol 2018 10.1186/s12874-018-0611-x 30453902
Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. Bmc Med Res Methodol. 2018. 10.1186/s12874-018-0611-x.30453902 10.1186/s12874-018-0611-x
39. Arksey H O'Malley L Scoping studies: towards a methodological framework Int J Soc Res Methodol 2005 8 1 19 32 10.1080/1364557032000119616
Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32. 10.1080/1364557032000119616.10.1080/1364557032000119616
40. Jackson N Waters E Guidelines for Systematic Reviews in Health P, Public Health T Criteria for the systematic review of health promotion and public health interventions Health Promot Int 2005 20 4 367 74 10.1093/heapro/dai022 16169885
Jackson N, Waters E, Guidelines for Systematic Reviews in Health P, Public Health T. Criteria for the systematic review of health promotion and public health interventions. Health Promot Int. 2005;20(4):367–74. 10.1093/heapro/dai022.16169885 10.1093/heapro/dai022
41. Behney C Medical technology—contributions to health care, the future of health in America 1989 Washington, D.C. U.S. Government Printing Office
Behney C. Medical technology—contributions to health care, the future of health in America. Washington, D.C.: U.S. Government Printing Office; 1989.
42. Abbass-Dick J Brolly M Huizinga J Newport A Xie F George S Designing an eHealth breastfeeding resource with indigenous families using a participatory design J Transcult Nurs 2018 29 5 480 488 10.1177/1043659617731818 29308703
Abbass-Dick J, Brolly M, Huizinga J, Newport A, Xie F, George S, et al. Designing an eHealth breastfeeding resource with indigenous families using a participatory design. J Transcult Nurs. 2018;29(5):480–8. 10.1177/1043659617731818.29308703 10.1177/1043659617731818
43. Abbass-Dick J Chyzzy B Newport A Huizinga J Xie F Designing an eHealth breastfeeding resource with young mothers using a participatory design J Transcult Nurs 2021 32 3 295 303 10.1177/1043659620957065 32909517
Abbass-Dick J, Chyzzy B, Newport A, Huizinga J, Xie F. Designing an eHealth breastfeeding resource with young mothers using a participatory design. J Transcult Nurs. 2021;32(3):295–303. 10.1177/1043659620957065.32909517 10.1177/1043659620957065
44. Ahmed R Toscos T Rohani Ghahari R Holden RJ Martin E Wagner S Visualization of cardiac implantable electronic device data for older adults using participatory design Appl Clin Inform 2019 10 4 707 718 10.1055/s-0039-1695794 31533172
Ahmed R, Toscos T, Rohani Ghahari R, Holden RJ, Martin E, Wagner S, et al. Visualization of cardiac implantable electronic device data for older adults using participatory design. Appl Clin Inform. 2019;10(4):707–18.31533172 10.1055/s-0039-1695794
45. Allin S Shepherd J Tomasone J Munce S Linassi G Hossain SN Participatory design of an online self-management tool for users with spinal cord injury: qualitative study JMIR Rehabil Assist Technol 2018 5 1 e6 10.2196/rehab.8158 29563075
Allin S, Shepherd J, Tomasone J, Munce S, Linassi G, Hossain SN, et al. Participatory design of an online self-management tool for users with spinal cord injury: qualitative study. JMIR Rehabil Assist Technol. 2018;5(1): e6. 10.2196/rehab.8158.29563075 10.2196/rehab.8158
46. Arevian AC O'Hora J Jones F Mango J Jones L Williams PG Participatory technology development to enhance community resilience Ethn Dis 2018 28 Suppl 2 493 502 10.18865/ed.28.S2.493 30202203
Arevian AC, O’Hora J, Jones F, Mango J, Jones L, Williams PG, et al. Participatory technology development to enhance community resilience. Ethn Dis. 2018;28(Suppl 2):493–502. 10.18865/ed.28.S2.493.30202203 10.18865/ed.28.S2.493
47. Arevian AC O'Hora J Rosser J Mango JD Miklowitz DJ Wells KB Patient and provider cocreation of mobile texting apps to support behavioral health: usability study JMIR Mhealth Uhealth 2020 8 7 e12655 10.2196/12655 32723714
Arevian AC, O’Hora J, Rosser J, Mango JD, Miklowitz DJ, Wells KB. Patient and provider cocreation of mobile texting apps to support behavioral health: usability study. JMIR Mhealth Uhealth. 2020;8(7): e12655. 10.2196/12655.32723714 10.2196/12655
48. Ceasar J Peters-Lawrence MH Mitchell V Powell-Wiley TM The communication, awareness, relationships and empowerment (C.A.R.E.) model: an effective tool for engaging urban communities in community-based participatory research Int J Environ Res Public Health 2017 10.3390/ijerph14111422 29160826
Ceasar J, Peters-Lawrence MH, Mitchell V, Powell-Wiley TM. The communication, awareness, relationships and empowerment (C.A.R.E.) model: an effective tool for engaging urban communities in community-based participatory research. Int J Environ Res Public Health. 2017. 10.3390/ijerph14111422.29160826 10.3390/ijerph14111422
49. De Cocker K Chastin SFM De Bourdeaudhuij I Imbo I Stragier J Cardon G Citizen science to communicate about public health messages: the reach of a playful online survey on sitting time and physical activity Health Commun 2019 34 7 720 725 10.1080/10410236.2018.1433955 29412005
De Cocker K, Chastin SFM, De Bourdeaudhuij I, Imbo I, Stragier J, Cardon G. Citizen science to communicate about public health messages: the reach of a playful online survey on sitting time and physical activity. Health Commun. 2019;34(7):720–5. 10.1080/10410236.2018.1433955.29412005 10.1080/10410236.2018.1433955
50. DeRenzi B, Dell N, Wacksman J, Lee S, Lesh N. Supporting Community Health Workers in India through Voice- and Web-Based Feedback. CHI Conference on Human Factors in Computing Systems; New York, NY, USA: Association for Computing Machinery; 2017. p. 2770–81.
51. Dewa LH Lawrence-Jones A Crandell C Jaques J Pickles K Lavelle M Reflections, impact and recommendations of a co-produced qualitative study with young people who have experience of mental health difficulties Health Expect 2021 24 Suppl 1 134 146 10.1111/hex.13088 32515538
Dewa LH, Lawrence-Jones A, Crandell C, Jaques J, Pickles K, Lavelle M, et al. Reflections, impact and recommendations of a co-produced qualitative study with young people who have experience of mental health difficulties. Health Expect. 2021;24(Suppl 1):134–46. 10.1111/hex.13088.32515538 10.1111/hex.13088
52. Doty JL Brady SS Monardez Popelka J Rietveld L Garcia-Huidobro D Doty MJ Designing a mobile app to enhance parenting skills of latinx parents: a community-based participatory approach JMIR Form Res 2020 4 1 e12618 10.2196/12618 32012034
Doty JL, Brady SS, Monardez Popelka J, Rietveld L, Garcia-Huidobro D, Doty MJ, et al. Designing a mobile app to enhance parenting skills of latinx parents: a community-based participatory approach. JMIR Form Res. 2020;4(1): e12618. 10.2196/12618.32012034 10.2196/12618
53. Edwards C Costello E Cassidy N Vick B Russell AM Use of the patientMpower app with home-based spirometry to monitor the symptoms and impact of fibrotic lung conditions: longitudinal observational study JMIR Mhealth Uhealth 2020 8 11 e16158 10.2196/16158 33216007
Edwards C, Costello E, Cassidy N, Vick B, Russell AM. Use of the patientMpower app with home-based spirometry to monitor the symptoms and impact of fibrotic lung conditions: longitudinal observational study. JMIR Mhealth Uhealth. 2020;8(11): e16158. 10.2196/16158.33216007 10.2196/16158
54. Elk R Emanuel L Hauser J Bakitas M Levkoff S Developing and testing the feasibility of a culturally based tele-palliative care consult based on the cultural values and preferences of Southern, Rural African American and White Community Members: a program by and for the community Health Equity 2020 4 1 52 83 10.1089/heq.2019.0120 32258958
Elk R, Emanuel L, Hauser J, Bakitas M, Levkoff S. Developing and testing the feasibility of a culturally based tele-palliative care consult based on the cultural values and preferences of Southern, Rural African American and White Community Members: a program by and for the community. Health Equity. 2020;4(1):52–83. 10.1089/heq.2019.0120.32258958 10.1089/heq.2019.0120
55. Erlingsdottir G, Rydenfalt C, Persson J, Johansson G. Conceptual Design of E-health Services by, and for Support of, Home Care Staff. Proceedings of the 20th Congress of the International Ergonomics Association (Iea 2018), Vol Vii: Ergonomics in Design, Design for All, Activity Theories for Work Analysis and Design, Affective Design. 2019; 824: 793–9. 10.1007/978-3-319-96071-5_83.
56. Evans C Turner K Suggs LS Occa A Juma A Blake H Developing a mHealth intervention to promote uptake of HIV testing among African communities in the conditions: a qualitative study Bmc Public Health 2016 10.1186/s12889-016-3278-4 27876023
Evans C, Turner K, Suggs LS, Occa A, Juma A, Blake H. Developing a mHealth intervention to promote uptake of HIV testing among African communities in the conditions: a qualitative study. Bmc Public Health. 2016. 10.1186/s12889-016-3278-4.27876023 10.1186/s12889-016-3278-4
57. Ferguson M Leighton P Brandreth M Wharrad H Development of a multimedia educational programme for first-time hearing aid users: a participatory design Int J Audiol 2018 57 8 600 609 10.1080/14992027.2018.1457803 29718733
Ferguson M, Leighton P, Brandreth M, Wharrad H. Development of a multimedia educational programme for first-time hearing aid users: a participatory design. Int J Audiol. 2018;57(8):600–9. 10.1080/14992027.2018.1457803.29718733 10.1080/14992027.2018.1457803
58. Ferri F, D’Andrea A, D’ulizia A, Grifoni P. Co-Creation of e-learning content: the case study of a MOOC on health and cyber-bullying. J Univ Comput Sci. 2020;26:200–19.
59. Fletcher S Mullett J Digital stories as a tool for health promotion and youth engagement Can J Public Health. 2016 107 2 e183-e7 10.17269/cjph.107.5266 27526216
Fletcher S, Mullett J. Digital stories as a tool for health promotion and youth engagement. Can J Public Health. 2016;107(2):e183–7.27526216 10.17269/cjph.107.5266
60. Fraco Z, Hooyer K, Rizia R, Kowser Patwary ABM, Armstrong M, Semaan B, et al. Crisis Warning Signs in mHealth for Military veterans: A Collaborative Design Approach. Proceedings of the ISCRAM 2016 Conference; Rio de Janeiro, Brazil: Tapia, Antunes, Bañuls, Moore and Porto de Albuquerque; 2016.
61. Frauenberger C, Kender K, Scheepmaker L, Werner K, Spiel K. Desiging Social Play Things. Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society; New York, NY, USA: Association for Computing Machinery; 2020. p. 1–12.
62. Garcia FE, Brandao RP, Mendes GCD, Neris VPD. Able to Create, Able to (Self-) improve: how an inclusive game framework fostered self-improvement through creation and play in alcohol and drugs rehabilitation. Human-computer interaction—interact 2019, Pt I. 2019;11746:337–58. 10.1007/978-3-030-29381-9_21.
63. Gardsten C Mortberg C Blomqvist K Designing an ICT self-management service: suggestions from persons with type 2 diabetes Health Technol-Ger 2017 7 2–3 197 206 10.1007/s12553-016-0176-9
Gardsten C, Mortberg C, Blomqvist K. Designing an ICT self-management service: suggestions from persons with type 2 diabetes. Health Technol-Ger. 2017;7(2–3):197–206. 10.1007/s12553-016-0176-9.10.1007/s12553-016-0176-9
64. Garmendia M, Epelde G, Zabala B, Sucre A, Etxeberría A, Cajaraville G, et al. ONKOVIS, Co-designed Oncology Follow-up Visualisation Tool. HEALTHINF; Prague, Czech Republic. 2019.
65. Giroux D Tremblay M Latulippe K Provencher V Poulin V Giguere A Promoting identification and use of aid resources by caregivers of seniors: co-design of an electronic health tool JMIR Aging 2019 2 2 e12314 10.2196/12314 31518284
Giroux D, Tremblay M, Latulippe K, Provencher V, Poulin V, Giguere A, et al. Promoting identification and use of aid resources by caregivers of seniors: co-design of an electronic health tool. JMIR Aging. 2019;2(2):e12314. 10.2196/12314.31518284 10.2196/12314
66. Gobl B, Hristova D, Jovicic S, Chevron MF, Slunecko T, Hlavacs H. Fostering Social Media Literacy through a Participatory Mixed-Methods Approach: Discussion of Workshop Findings. Ieee Int Conf Seriou. 2019
67. Grant C Widnall E Cross L Simonoff E Downs J Informing the development of an E-platform for monitoring wellbeing in schools: involving young people in a co-design process Res Involv Engagem 2020 6 51 10.1186/s40900-020-00219-0 32908677
Grant C, Widnall E, Cross L, Simonoff E, Downs J. Informing the development of an E-platform for monitoring wellbeing in schools: involving young people in a co-design process. Res Involv Engagem. 2020;6:51. 10.1186/s40900-020-00219-0.32908677 10.1186/s40900-020-00219-0
68. Haufe M Peek STM Luijkx KG Matching gerontechnologies to independent-living seniors' individual needs: development of the GTM tool BMC Health Serv Res 2019 19 1 26 10.1186/s12913-018-3848-5 30634971
Haufe M, Peek STM, Luijkx KG. Matching gerontechnologies to independent-living seniors’ individual needs: development of the GTM tool. BMC Health Serv Res. 2019;19(1):26. 10.1186/s12913-018-3848-5.30634971 10.1186/s12913-018-3848-5
69. Hernandez HA Khan A Fay L Roy JS Biddiss E Force resistance training in hand grasp and arm therapy: feasibility of a low-cost videogame controller Games Health J 2018 7 4 277 287 10.1089/g4h.2017.0193 30106641
Hernandez HA, Khan A, Fay L, Roy JS, Biddiss E. Force resistance training in hand grasp and arm therapy: feasibility of a low-cost videogame controller. Games Health J. 2018;7(4):277–87. 10.1089/g4h.2017.0193.30106641 10.1089/g4h.2017.0193
70. Hill ME Bodnar P Fenton R Mason B Bandoh G Teach our children: stroke education for indigenous children, first nations, Ontario, Canada, 2009–2012 Prev Chronic Dis 2017 14 E68 10.5888/pcd14.160506 28817789
Hill ME, Bodnar P, Fenton R, Mason B, Bandoh G. Teach our children: stroke education for indigenous children, first nations, Ontario, Canada, 2009–2012. Prev Chronic Dis. 2017;14:E68. 10.5888/pcd14.160506.28817789 10.5888/pcd14.160506
71. Hiratsuka VY Moore L Avey JP Dirks LG Beach BD Dillard DA An internet-based therapeutic tool for American Indian/Alaska native adults with posttraumatic stress disorder: user testing and developmental feasibility study JMIR Form Res 2019 3 4 e13682 10.2196/13682 31719027
Hiratsuka VY, Moore L, Avey JP, Dirks LG, Beach BD, Dillard DA, et al. An internet-based therapeutic tool for American Indian/Alaska native adults with posttraumatic stress disorder: user testing and developmental feasibility study. JMIR Form Res. 2019;3(4): e13682. 10.2196/13682.31719027 10.2196/13682
72. Hong MK, Lakshmi U, Olson TA, Wilcox L. Visual ODLs: Co-Designing Patient-Generated Observations of Daily Living to Support Data-Driven Conversations in Pediatric Care. Proceedings of the 2018 Chi Conference on Human Factors in Computing Systems (Chi 2018). 2018. 10.1145/3173574.3174050
73. Jarke J, Gerhard U, Kubicek H. Co-creating digital public services with older citizens: Challenges and opportunities. In: David K, Geihs K, Lange M, Stumme G, editors. INFORMATIK: 50 Jahre Gesellschaft für Informatik – Informatik für Gesellschaft; Kassel, Germany: Bonn: Gesellschaft für Informatik e.V.; 2019. p. 51–64.
74. Jessen S Mirkovic J Nes LS MyStrengths, a strengths-focused mobile health tool: participatory design and development JMIR Form Res 2020 4 7 e18049 10.2196/18049 32706651
Jessen S, Mirkovic J, Nes LS. MyStrengths, a strengths-focused mobile health tool: participatory design and development. JMIR Form Res. 2020;4(7): e18049. 10.2196/18049.32706651 10.2196/18049
75. Jessen S Mirkovic J Ruland CM Creating gameful design in mHealth: a participatory co-design approach JMIR Mhealth Uhealth 2018 6 12 e11579 10.2196/11579 30552080
Jessen S, Mirkovic J, Ruland CM. Creating gameful design in mHealth: a participatory co-design approach. JMIR Mhealth Uhealth. 2018;6(12): e11579. 10.2196/11579.30552080 10.2196/11579
76. Jo E, Toombs AL, Gray CM, Hong H. Understanding Parenting Stress through Co-designed Self-Trackers. Proceedings of the 2020 Chi Conference on Human Factors in Computing Systems (Chi'20). 2020. 10.1145/3313831.3376359.
77. Jorgensen TS Skougaard M Taylor PC Asmussen HC Lee A Klokker L The Parker model: applying a qualitative three-step approach to optimally utilize input from stakeholders when introducing new device technologies in the management of chronic rheumatic diseases Patient 2018 11 5 515 526 10.1007/s40271-018-0306-8 29948962
Jorgensen TS, Skougaard M, Taylor PC, Asmussen HC, Lee A, Klokker L, et al. The Parker model: applying a qualitative three-step approach to optimally utilize input from stakeholders when introducing new device technologies in the management of chronic rheumatic diseases. Patient. 2018;11(5):515–26. 10.1007/s40271-018-0306-8.29948962 10.1007/s40271-018-0306-8
78. Kanstrup AM, Bertelsen P. Bringing new voices to design of exercise technology: participatory design with vulnerable young adults. Participatory Design Conference; New York, NY, USA: Association for Computing Machinery; 2016. p. 121–30.
79. Kearns A Kelly H Pitt I Rating experience of ICT-delivered aphasia rehabilitation: co-design of a feedback questionnaire Aphasiology 2020 34 3 319 342 10.1080/02687038.2019.1649913
Kearns A, Kelly H, Pitt I. Rating experience of ICT-delivered aphasia rehabilitation: co-design of a feedback questionnaire. Aphasiology. 2020;34(3):319–42. 10.1080/02687038.2019.1649913.10.1080/02687038.2019.1649913
80. Kenning G Treadaway C Designing for Dementia: iterative grief and transitional objects Des Issues 2018 34 1 42 53 10.1162/DESI_a_00475
Kenning G, Treadaway C. Designing for Dementia: iterative grief and transitional objects. Des Issues. 2018;34(1):42–53. 10.1162/DESI_a_00475.10.1162/DESI_a_00475
81. Kildea J Battista J Cabral B Hendren L Herrera D Hijal T Design and development of a person-centered patient portal using participatory stakeholder co-design J Med Internet Res 2019 10.2196/11371 30741643
Kildea J, Battista J, Cabral B, Hendren L, Herrera D, Hijal T, et al. Design and development of a person-centered patient portal using participatory stakeholder co-design. J Med Internet Res. 2019. 10.2196/11371.30741643 10.2196/11371
82. Kim SI Jo E Ryu M Cha I Kim YH Yoo H Toward becoming a better self: understanding self-tracking experiences of adolescents with autism spectrum disorder using custom trackers Int Conf Per Comp 2019 10.1145/3329189.3329209
Kim SI, Jo E, Ryu M, Cha I, Kim YH, Yoo H, et al. Toward becoming a better self: understanding self-tracking experiences of adolescents with autism spectrum disorder using custom trackers. Int Conf Per Comp. 2019. 10.1145/3329189.3329209.10.1145/3329189.3329209
83. LaMonica HM Milton A Braunstein K Rowe SC Ottavio A Jackson T Technology-enabled solutions for Australian mental health services reform: impact evaluation JMIR Form Res 2020 4 11 e18759 10.2196/18759 33211024
LaMonica HM, Milton A, Braunstein K, Rowe SC, Ottavio A, Jackson T, et al. Technology-enabled solutions for Australian mental health services reform: impact evaluation. JMIR Form Res. 2020;4(11): e18759. 10.2196/18759.33211024 10.2196/18759
84. Lan Hing Ting K Dessinger G Voilmy D Examining usage to ensure utility: co-design of a tool for fall prevention Irbm 2020 41 5 286 93 10.1016/j.irbm.2020.03.001
Lan Hing Ting K, Dessinger G, Voilmy D. Examining usage to ensure utility: co-design of a tool for fall prevention. Irbm. 2020;41(5):286–93. 10.1016/j.irbm.2020.03.001.10.1016/j.irbm.2020.03.001
85. Larsen SB Sorensen NS Petersen MG Kjeldsen GF Towards a shared service centre for telemedicine: telemedicine in Denmark, and a possible way forward Health Informatics J 2016 22 4 815 827 10.1177/1460458215592042 26261216
Larsen SB, Sorensen NS, Petersen MG, Kjeldsen GF. Towards a shared service centre for telemedicine: telemedicine in Denmark, and a possible way forward. Health Informatics J. 2016;22(4):815–27. 10.1177/1460458215592042.26261216 10.1177/1460458215592042
86. Latulippe K Hamel C Giroux D Co-design to support the development of inclusive eHealth tools for caregivers of functionally dependent older persons: social justice design J Med Internet Res 2020 22 11 e18399 10.2196/18399 33164905
Latulippe K, Hamel C, Giroux D. Co-design to support the development of inclusive eHealth tools for caregivers of functionally dependent older persons: social justice design. J Med Internet Res. 2020;22(11): e18399. 10.2196/18399.33164905 10.2196/18399
87. Latulippe K Hamel C Giroux D Integration of conversion factors for the development of an inclusive ehealth tool with caregivers of functionally dependent older persons: social justice design JMIR Hum Factors 2020 7 3 e18120 10.2196/18120 32845242
Latulippe K, Hamel C, Giroux D. Integration of conversion factors for the development of an inclusive ehealth tool with caregivers of functionally dependent older persons: social justice design. JMIR Hum Factors. 2020;7(3): e18120. 10.2196/18120.32845242 10.2196/18120
88. Latulippe K Tremblay M Poulin V Provencher V Giguere AM Sevigny A Prioritizing the needs of caregivers of older adults to support their help-seeking process as a first step to developing an eHealth tool: the technique for research of information by animation of a group of experts (TRIAGE) method JMIR Aging 2019 2 1 e12271 10.2196/12271 31518269
Latulippe K, Tremblay M, Poulin V, Provencher V, Giguere AM, Sevigny A, et al. Prioritizing the needs of caregivers of older adults to support their help-seeking process as a first step to developing an eHealth tool: the technique for research of information by animation of a group of experts (TRIAGE) method. JMIR Aging. 2019;2(1): e12271. 10.2196/12271.31518269 10.2196/12271
89. Lee HR Sabanovic S Chang WL Nagata S Piatt J Bennett C Steps Toward participatory design of social robots: mutual learning with older adults with depression Acmieee Int Conf Hum 2017 10.1145/2909824.3020237
Lee HR, Sabanovic S, Chang WL, Nagata S, Piatt J, Bennett C, et al. Steps Toward participatory design of social robots: mutual learning with older adults with depression. Acmieee Int Conf Hum. 2017. 10.1145/2909824.3020237.10.1145/2909824.3020237
90. Leeming G Thew S DataWell: public involvement in the development of a federated platform for shared patient records in Greater Manchester UK Stud Health Technol 2017 244 48 52 10.3233/978-1-61499-824-2-48
Leeming G, Thew S. DataWell: public involvement in the development of a federated platform for shared patient records in Greater Manchester. UK Stud Health Technol. 2017;244:48–52. 10.3233/978-1-61499-824-2-48.10.3233/978-1-61499-824-2-48
91. Leong TW Johnston B Co-design and robots: a case study of a robot dog for aging people Lect Notes Artif Int 2016 9979 702 711 10.1007/978-3-319-47437-3_69
Leong TW, Johnston B. Co-design and robots: a case study of a robot dog for aging people. Lect Notes Artif Int. 2016;9979:702–11. 10.1007/978-3-319-47437-3_69.10.1007/978-3-319-47437-3_69
92. Leslie M Khayatzadeh-Mahani A MacKean G Recruitment of caregivers into health services research: lessons from a user-centred design study Res Involv Engagem 2019 5 17 10.1186/s40900-019-0150-6 31139432
Leslie M, Khayatzadeh-Mahani A, MacKean G. Recruitment of caregivers into health services research: lessons from a user-centred design study. Res Involv Engagem. 2019;5:17. 10.1186/s40900-019-0150-6.31139432 10.1186/s40900-019-0150-6
93. Limaye NP Rivas-Nieto AC Carcamo CP Blas MM Nuestras historias- designing a novel digital story intervention through participatory methods to improve maternal and child health in the Peruvian Amazon PLoS ONE 2018 13 11 e0205673 10.1371/journal.pone.0205673 30396200
Limaye NP, Rivas-Nieto AC, Carcamo CP, Blas MM. Nuestras historias- designing a novel digital story intervention through participatory methods to improve maternal and child health in the Peruvian Amazon. PLoS ONE. 2018;13(11): e0205673. 10.1371/journal.pone.0205673.30396200 10.1371/journal.pone.0205673
94. Livingood WC Monticalvo D Bernhardt JM Wells KT Harris T Kee K Engaging adolescents through participatory and qualitative research methods to develop a digital communication intervention to reduce adolescent obesity Health Educ Behav 2017 44 4 570 580 10.1177/1090198116677216 27811164
Livingood WC, Monticalvo D, Bernhardt JM, Wells KT, Harris T, Kee K, et al. Engaging adolescents through participatory and qualitative research methods to develop a digital communication intervention to reduce adolescent obesity. Health Educ Behav. 2017;44(4):570–80.27811164 10.1177/1090198116677216
95. Lo B Zhang T Leung K Mehta R Kuziemsky C Booth RG Identifying best approaches for engaging patients and family members in health informatics initiatives: a case study of the group priority sort technique Res Involv Engagem 2020 6 25 10.1186/s40900-020-00203-8 32477591
Lo B, Zhang T, Leung K, Mehta R, Kuziemsky C, Booth RG, et al. Identifying best approaches for engaging patients and family members in health informatics initiatives: a case study of the group priority sort technique. Res Involv Engagem. 2020;6:25. 10.1186/s40900-020-00203-8.32477591 10.1186/s40900-020-00203-8
96. Lockerbie J Maiden N Modelling the quality of life goals of people living with dementia Inf Syst 2022 10.1016/j.is.2020.101578
Lockerbie J, Maiden N. Modelling the quality of life goals of people living with dementia. Inf Syst. 2022. 10.1016/j.is.2020.101578.10.1016/j.is.2020.101578
97. Lu Y Valk C Steenbakkers J Bekker T Visser T Proctor G Can technology adoption for older adults be co-created? Gerontechnology 2017 16 3 151 159 10.4017/gt.2017.16.3.004.00
Lu Y, Valk C, Steenbakkers J, Bekker T, Visser T, Proctor G, et al. Can technology adoption for older adults be co-created? Gerontechnology. 2017;16(3):151–9. 10.4017/gt.2017.16.3.004.00.10.4017/gt.2017.16.3.004.00
98. Maertens JA Jimenez-Zambrano AM Albright K Dempsey AF Using community engagement to develop a web-based intervention for Latinos about the HPV vaccine J Health Commun 2017 22 4 285 293 10.1080/10810730.2016.1275890 28276945
Maertens JA, Jimenez-Zambrano AM, Albright K, Dempsey AF. Using community engagement to develop a web-based intervention for Latinos about the HPV vaccine. J Health Commun. 2017;22(4):285–93. 10.1080/10810730.2016.1275890.28276945 10.1080/10810730.2016.1275890
99. Martin A Caon M Adorni F Andreoni G Ascolese A Atkinson S A mobile phone intervention to improve obesity-related health behaviors of adolescents across Europe: iterative co-design and feasibility study Jmir Mhealth Uhealth 2020 10.2196/14118 32130179
Martin A, Caon M, Adorni F, Andreoni G, Ascolese A, Atkinson S, et al. A mobile phone intervention to improve obesity-related health behaviors of adolescents across Europe: iterative co-design and feasibility study. Jmir Mhealth Uhealth. 2020. 10.2196/14118.32130179 10.2196/14118
100. Matthews M, Abdullah S, Gay G, Choudhury T. Detecting and Capitalizing on Physiological Dimensions of Psychiatric Illness. Physics: Proceedings of the 3rd International Conference on Physiological Computing Systems. 2016:98–104. 10.5220/0005952600980104.
101. Mburu CW, Wardle CJ, Joolay Y, Densmore M. Co-designing with Mothers and Neonatal Unit Staff: Use of Technology to Support Mothers of Preterm Infants. Proceedings of the Second African Conference for Human Computer Interaction: Thriving Communities (Africhi). 2018:118–27. 10.1145/3283458.3283487
102. McMillan B Fox S Lyons M Bourke S Mistry M Ruddock A Using patient and public involvement to improve the research design and funding application for a project aimed at fostering a more collaborative approach to the NHS health check: the CaVIAR project (better care Via improved access to records) Res Involve Engage 2018 10.1186/s40900-018-0101-7
McMillan B, Fox S, Lyons M, Bourke S, Mistry M, Ruddock A, et al. Using patient and public involvement to improve the research design and funding application for a project aimed at fostering a more collaborative approach to the NHS health check: the CaVIAR project (better care Via improved access to records). Res Involve Engage. 2018. 10.1186/s40900-018-0101-7.10.1186/s40900-018-0101-7
103. McNaney R, Vines J, Mercer J, Mexter L, Welsh D, Young T. DemYouth: Co-Designing and Enacting Tools to Support Young People's Engagement with People with Dementia. Proceedings of the 2017 Acm Sigchi Conference on Human Factors in Computing Systems (Chi'17). 2017: 1313–25. 10.1145/3025453.3025558
104. Miller AD Pollack AH Pratt W Bursting the information bubble: identifying opportunities for pediatric patient-centered technology AMIA Annu Symp Proc 2016 2016 894 903 28269886
Miller AD, Pollack AH, Pratt W. Bursting the information bubble: identifying opportunities for pediatric patient-centered technology. AMIA Annu Symp Proc. 2016;2016:894–903.28269886
105. Mrklas KJ Barber T Campbell-Scherer D Green LA Li LC Marlett N Co-design in the development of a mobile health app for the management of knee osteoarthritis by patients and physicians: qualitative study JMIR Mhealth Uhealth 2020 8 7 e17893 10.2196/17893 32673245
Mrklas KJ, Barber T, Campbell-Scherer D, Green LA, Li LC, Marlett N, et al. Co-design in the development of a mobile health app for the management of knee osteoarthritis by patients and physicians: qualitative study. JMIR Mhealth Uhealth. 2020;8(7): e17893. 10.2196/17893.32673245 10.2196/17893
106. Naeemabadi M Sondergaard JH Klastrup A Schlunsen AP Lauritsen REK Hansen J Development of an individualized asynchronous sensor-based telerehabilitation program for patients undergoing total knee replacement: participatory design Health Informatics J 2020 26 4 2492 2511 10.1177/1460458220909779 32175788
Naeemabadi M, Sondergaard JH, Klastrup A, Schlunsen AP, Lauritsen REK, Hansen J, et al. Development of an individualized asynchronous sensor-based telerehabilitation program for patients undergoing total knee replacement: participatory design. Health Informatics J. 2020;26(4):2492–511. 10.1177/1460458220909779.32175788 10.1177/1460458220909779
107. Payton FC Cultures of participationfor students, by students Inform Syst J 2016 26 4 319 338 10.1111/isj.12086
Payton FC. Cultures of participationfor students, by students. Inform Syst J. 2016;26(4):319–38. 10.1111/isj.12086.10.1111/isj.12086
108. Rabba AS Dissanayake C Barbaro J Development of a web-based resource for parents of young children newly diagnosed with autism: participatory research design Jmir Pediatr Parent 2020 10.2196/15786 32996890
Rabba AS, Dissanayake C, Barbaro J. Development of a web-based resource for parents of young children newly diagnosed with autism: participatory research design. Jmir Pediatr Parent. 2020. 10.2196/15786.32996890 10.2196/15786
109. Rai T Morton K Roman C Doogue R Rice C Williams M Optimizing a digital intervention for managing blood pressure in stroke patients using a diverse sample: integrating the person-based approach and patient and public involvement Health Expect 2021 24 2 327 340 10.1111/hex.13173 33316120
Rai T, Morton K, Roman C, Doogue R, Rice C, Williams M, et al. Optimizing a digital intervention for managing blood pressure in stroke patients using a diverse sample: integrating the person-based approach and patient and public involvement. Health Expect. 2021;24(2):327–40. 10.1111/hex.13173.33316120 10.1111/hex.13173
110. Rathnayake S Moyle W Jones C Calleja P Co-design of an mHealth application for family caregivers of people with dementia to address functional disability care needs Inform Health Soc Ca 2021 46 1 1 17 10.1080/17538157.2020.1793347
Rathnayake S, Moyle W, Jones C, Calleja P. Co-design of an mHealth application for family caregivers of people with dementia to address functional disability care needs. Inform Health Soc Ca. 2021;46(1):1–17. 10.1080/17538157.2020.1793347.10.1080/17538157.2020.1793347
111. Reade S Spencer K Sergeant JC Sperrin M Schultz DM Ainsworth J Cloudy with a chance of pain: engagement and subsequent attrition of daily data entry in a smartphone pilot study tracking weather, disease severity, and physical activity in patients with rheumatoid arthritis JMIR Mhealth Uhealth 2017 5 3 e37 10.2196/mhealth.6496 28341616
Reade S, Spencer K, Sergeant JC, Sperrin M, Schultz DM, Ainsworth J, et al. Cloudy with a chance of pain: engagement and subsequent attrition of daily data entry in a smartphone pilot study tracking weather, disease severity, and physical activity in patients with rheumatoid arthritis. JMIR Mhealth Uhealth. 2017;5(3):e37. 10.2196/mhealth.6496.28341616 10.2196/mhealth.6496
112. Russ S Latif Z Hazell AL Ogunmuyiwa H Tapper J Wachuku-King S A smartphone app designed to empower patients to contribute toward safer surgical care: community-based evaluation using a participatory approach JMIR Mhealth Uhealth 2020 8 1 e12859 10.2196/12859 31958067
Russ S, Latif Z, Hazell AL, Ogunmuyiwa H, Tapper J, Wachuku-King S, et al. A smartphone app designed to empower patients to contribute toward safer surgical care: community-based evaluation using a participatory approach. JMIR Mhealth Uhealth. 2020;8(1): e12859. 10.2196/12859.31958067 10.2196/12859
113. Sin J Henderson C Woodham LA Sese Hernandez A Gillard S A multicomponent eHealth intervention for family carers for people affected by psychosis: a coproduced design and build study J Med Internet Res 2019 21 8 e14374 10.2196/14374 31389333
Sin J, Henderson C, Woodham LA, Sese Hernandez A, Gillard S. A multicomponent eHealth intervention for family carers for people affected by psychosis: a coproduced design and build study. J Med Internet Res. 2019;21(8): e14374. 10.2196/14374.31389333 10.2196/14374
114. Skovlund PC Nielsen BK Thaysen HV Schmidt H Finset A Hansen KA The impact of patient involvement in research: a case study of the planning, conduct and dissemination of a clinical, controlled trial Res Involv Engage 2020 6 43 10.1186/s40900-020-00214-5
Skovlund PC, Nielsen BK, Thaysen HV, Schmidt H, Finset A, Hansen KA, et al. The impact of patient involvement in research: a case study of the planning, conduct and dissemination of a clinical, controlled trial. Res Involv Engage. 2020;6:43. 10.1186/s40900-020-00214-5.10.1186/s40900-020-00214-5
115. Solem IKL Varsi C Eide H Kristjansdottir OB Borosund E Schreurs KMG A user-centered approach to an evidence-based electronic health pain management intervention for people with chronic pain: design and development of EPIO J Med Internet Res 2020 10.2196/15889
Solem IKL, Varsi C, Eide H, Kristjansdottir OB, Borosund E, Schreurs KMG, et al. A user-centered approach to an evidence-based electronic health pain management intervention for people with chronic pain: design and development of EPIO. J Med Internet Res. 2020. 10.2196/15889.10.2196/15889
116. Synnot AJ Hawkins M Merner BA Summers MP Filippini G Osborne RH Producing an evidence-based treatment information website in partnership with people affected by multiple sclerosis Health Sci Rep-Us 2018 10.1002/hsr2.24
Synnot AJ, Hawkins M, Merner BA, Summers MP, Filippini G, Osborne RH, et al. Producing an evidence-based treatment information website in partnership with people affected by multiple sclerosis. Health Sci Rep-Us. 2018. 10.1002/hsr2.24.10.1002/hsr2.24
117. Tsvyatkova D Storni C Designing an educational interactive eBook for newly diagnosed children with type 1 diabetes: mapping a new design space Int J Child-Comput Interact 2019 19 1 18 10.1016/j.ijcci.2018.10.001
Tsvyatkova D, Storni C. Designing an educational interactive eBook for newly diagnosed children with type 1 diabetes: mapping a new design space. Int J Child-Comput Interact. 2019;19:1–18.10.1016/j.ijcci.2018.10.001
118. Tuckett AG Freeman A Hetherington S Gardiner PA King AC Burnie Brae Citizen S Older adults using our voice citizen science to create change in their neighborhood environment Int J Environ Res Public Health 2018 10.3390/ijerph15122685 30487444
Tuckett AG, Freeman A, Hetherington S, Gardiner PA, King AC, Burnie Brae Citizen S. Older adults using our voice citizen science to create change in their neighborhood environment. Int J Environ Res Public Health. 2018. 10.3390/ijerph15122685.30487444 10.3390/ijerph15122685
119. Vacca R. Brokering Data: Co-Designing Technology with Latina Teens to Support Communication with Parents Leveraging Cultural Practices of Latinx Youth through Co-Design. Proceedings of Acm Interaction Design and Children (Idc 2019). 2019; 10.1145/3311927.3323142
120. Vallentin-Holbech L Guldager JD Dietrich T Rundle-Thiele S Majgaard G Lyk P Co-creating a virtual alcohol prevention simulation with young people Int J Env Res Pub He 2020 10.3390/ijerph17031097
Vallentin-Holbech L, Guldager JD, Dietrich T, Rundle-Thiele S, Majgaard G, Lyk P, et al. Co-creating a virtual alcohol prevention simulation with young people. Int J Env Res Pub He. 2020. 10.3390/ijerph17031097.10.3390/ijerph17031097
121. Vilarinho T Floch J Stav E Co-designing a mHealth application for self-management of cystic fibrosis Lect Notes Comput Sc 2017 10515 3 22 10.1007/978-3-319-67687-6_1
Vilarinho T, Floch J, Stav E. Co-designing a mHealth application for self-management of cystic fibrosis. Lect Notes Comput Sc. 2017;10515:3–22. 10.1007/978-3-319-67687-6_1.10.1007/978-3-319-67687-6_1
122. Wikman A Kukkola L Borjesson H Cernvall M Woodford J Gronqvist H Development of an internet-administered cognitive behavior therapy program (ENGAGE) for parents of children previously treated for cancer: participatory action research approach J Med Internet Res 2018 10.2196/jmir.9457 29669710
Wikman A, Kukkola L, Borjesson H, Cernvall M, Woodford J, Gronqvist H, et al. Development of an internet-administered cognitive behavior therapy program (ENGAGE) for parents of children previously treated for cancer: participatory action research approach. J Med Internet Res. 2018. 10.2196/jmir.9457.29669710 10.2196/jmir.9457
123. Yarosh S Schueller SM “Happiness inventors”: informing positive computing technologies through participatory design with children J Med Internet Res 2017 10.2196/jmir.6822 28096066
Yarosh S, Schueller SM. “Happiness inventors”: informing positive computing technologies through participatory design with children. J Med Internet Res. 2017. 10.2196/jmir.6822.28096066 10.2196/jmir.6822
124. Spinuzzi C The methodology of participatory design Tech Commun-Stc 2005 52 2 163 174
Spinuzzi C. The methodology of participatory design. Tech Commun-Stc. 2005;52(2):163–74.
125. McIntyre A Participatory action research 2007 Los Angeles, CA Sage Publications
McIntyre A. Participatory action research. Los Angeles, CA: Sage Publications; 2007.
126. Kindon S Pain R Kesby M Participatory action research approaches and methods: connecting people, participationand place 2008 New York Routhledge
Kindon S, Pain R, Kesby M. Participatory action research approaches and methods: connecting people, participationand place. New York: Routhledge; 2008.
127. Baum F MacDougall C Smith D Participatory action research J Epidemiol Commun Health 2006 60 10 854 857 10.1136/jech.2004.028662
Baum F, MacDougall C, Smith D. Participatory action research. J Epidemiol Commun Health. 2006;60(10):854–7. 10.1136/jech.2004.028662.10.1136/jech.2004.028662
128. Druin A, editor Cooperative Inquiry: Developing New Technologies for Children with Children. SIGCHI Conference on Human Factors in Computing Systems; 1999: ACM, New York, NY, USA.
129. Makhaeva J, Frauenberger C, Spiel K. Creating creative spaces for co-designing with Autistic children—the concept of a “Handlungsspielraum”. 14th Participatory Design; 2016: ACM Press, Aarhus, Denmark.
130. Irizarry T DeVito DA Curran CR Patient portals and patient engagement: a state of the science review J Med Internet Res 2015 17 6 e148 10.2196/jmir.4255 26104044
Irizarry T, DeVito DA, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res. 2015;17(6): e148. 10.2196/jmir.4255.26104044 10.2196/jmir.4255
131. Bate P Robert G Experience-based design: from redesigning the system around the patient to co-designing services with the patient Qual Saf Health Care 2006 15 5 307 310 10.1136/qshc.2005.016527 17074863
Bate P, Robert G. Experience-based design: from redesigning the system around the patient to co-designing services with the patient. Qual Saf Health Care. 2006;15(5):307–10. 10.1136/qshc.2005.016527.17074863 10.1136/qshc.2005.016527
132. Ekman I Swedberg K Taft C Lindseth A Norberg A Brink E Person-centered care–ready for prime time Eur J Cardiovasc Nurs 2011 10 4 248 251 10.1016/j.ejcnurse.2011.06.008 21764386
Ekman I, Swedberg K, Taft C, Lindseth A, Norberg A, Brink E, et al. Person-centered care–ready for prime time. Eur J Cardiovasc Nurs. 2011;10(4):248–51. 10.1016/j.ejcnurse.2011.06.008.21764386 10.1016/j.ejcnurse.2011.06.008
133. Beck K, Beedle M, van Bennekum A, Cockburn A, Cunningham W, Fowler M, et al. Principles behind the agile manifesto 2001. http://agilemanifesto.org/principles.html. Accessed Sep 2022.
134. Bonney R Cooper C Ballard H The theory and practice of citizen science: launching a new journal Citizen Sci 2016 1 1 1 10.5334/cstp.65
Bonney R, Cooper C, Ballard H. The theory and practice of citizen science: launching a new journal. Citizen Sci. 2016;1(1):1. 10.5334/cstp.65.10.5334/cstp.65
135. Healey J Conlon CM Malama K Hobson R Kaharuza F Kekitiinwa A Sustainability and scale of the saving mothers, giving life approach in Uganda and Zambia Glob Health Sci Pract 2019 7 Suppl 1 S188 S206 10.9745/GHSP-D-18-00265 30867217
Healey J, Conlon CM, Malama K, Hobson R, Kaharuza F, Kekitiinwa A, et al. Sustainability and scale of the saving mothers, giving life approach in Uganda and Zambia. Glob Health Sci Pract. 2019;7(Suppl 1):S188–206. 10.9745/GHSP-D-18-00265.30867217 10.9745/GHSP-D-18-00265
136. Hodes R Doubt J Toska E Vale B Zungu N Cluver L The stuff that dreams are made of: HIV-positive adolescents' aspirations for development J Int AIDS Soc 2018 21 Suppl 1 10.1002/jia2.25057
Hodes R, Doubt J, Toska E, Vale B, Zungu N, Cluver L. The stuff that dreams are made of: HIV-positive adolescents’ aspirations for development. J Int AIDS Soc. 2018;21(Suppl):1. 10.1002/jia2.25057.10.1002/jia2.25057
137. Iles IA Nan XL Ma ZX Feldman R Butler J Wang MQ Self-affirmation and defensive processing of graphic cigarette warning labels by African American smokers: a community-based study Health Educ J 2019 78 3 301 314 10.1177/0017896918805129
Iles IA, Nan XL, Ma ZX, Feldman R, Butler J, Wang MQ, et al. Self-affirmation and defensive processing of graphic cigarette warning labels by African American smokers: a community-based study. Health Educ J. 2019;78(3):301–14. 10.1177/0017896918805129.10.1177/0017896918805129
138. Kalokhe AS Iyer S Katendra T Gadhe K Kolhe AR Paranjape A Primary prevention of intimate partner violence among recently married dyads residing in the slums of Pune, India: development and rationale for a dyadic intervention Jmir Res Protoc 2019 10.2196/11533 30664483
Kalokhe AS, Iyer S, Katendra T, Gadhe K, Kolhe AR, Paranjape A, et al. Primary prevention of intimate partner violence among recently married dyads residing in the slums of Pune, India: development and rationale for a dyadic intervention. Jmir Res Protoc. 2019. 10.2196/11533.30664483 10.2196/11533
139. Greenwood DA Litchman ML Ng AH Gee PM Young HM Ferrer M Development of the intercultural diabetes online community research council: codesign and social media processes J Diabetes Sci Technol 2019 13 2 176 186 10.1177/1932296818818455 30614252
Greenwood DA, Litchman ML, Ng AH, Gee PM, Young HM, Ferrer M, et al. Development of the intercultural diabetes online community research council: codesign and social media processes. J Diabetes Sci Technol. 2019;13(2):176–86. 10.1177/1932296818818455.30614252 10.1177/1932296818818455
140. Ostrow L Penney D Stuart E Leaf PJ Web-based survey data collection with peer support and advocacy organizations: implications of participatory methods Prog Commun Health Partnersh 2017 11 1 45 52 10.1353/cpr.2017.0006
Ostrow L, Penney D, Stuart E, Leaf PJ. Web-based survey data collection with peer support and advocacy organizations: implications of participatory methods. Prog Commun Health Partnersh. 2017;11(1):45–52. 10.1353/cpr.2017.0006.10.1353/cpr.2017.0006
141. Bratteteig T, Wagner I. What is a participatory design result? 14th Participatory Design Conference; 2016: ACM Press, Aarhus, Denmark.
