
==== Front
Perm J
tpj
tpj
The Permanente Journal
1552-5767
1552-5775
The Permanente Press

39113492
10.7812/TPP/24.012
TPJ-24-012
Review Article
Diversity in Combating Complex, Chronic, and Costly Conditions Using a Learning Health System Approach: A Scoping Review
https://orcid.org/0000-0002-8363-2446
Davis Rebekah A MPH 1
Sine Kathryn MD 2
Burguera-Couce Ella BA 2
Ahmad Jabeen PhD, MPH 3
https://orcid.org/0000-0002-7399-622X
Oliver Brant J PhD, MS, MPH, FNP-BC, PMHNP-BC 1 3 4 5
1 The Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth, Hanover, NH, USA
2 The Warren Alpert Medical School of Brown University, Providence, RI, USA
3 Chronic Health Improvement Research Program at Dartmouth Health, Department of Community & Family Medicine, Dartmouth Health, Lebanon, NH, USA
4 Department of Psychiatry, Dartmouth Health, Lebanon, NH, USA
5 Office of Care Experience, the Value Institute, Dartmouth Health, Lebanon, NH, USA
Brant J Oliver, PhD, MS, MPH, FNP-BC, PMHNP-BC brant.j.oliver@dartmouth.edu
Supplementary Materials: Supplemental Material is available at: www.thepermanentejournal.org/files/2024/24.012supp.pdf

2024
08 8 2024
28 3 245261
© 2024 The Authors.
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Published by The Permanente Federation LLC under the terms of the CC BY-NC-ND 4.0 license https://creativecommons.org/licenses/by-nc-nd/4.0/.

Abstract

Introduction

The purpose of this scoping review was to investigate in the literature how a learning health system (LHS) can be implemented in cases of complex, costly, chronic (3C) conditions.

Methods

A scoping review of literature published in English since 2007 was conducted using Medline, Cumulative Index to Nursing and Allied Health Literature, and Scopus. Two authors screened the resulting articles and two authors extracted study details on the structure, process, and outcome of each LHS. Eligibility criteria included studies of LHSs that focused on populations experiencing a complex chronic health condition. A narrative synthesis of data was conducted using deductive qualitative methods.

Results

Application of the authors’ search strategy resulted in 656 publications that were analyzed for this review. The authors included 17 studies that focused on 13 LHSs. The structure of the LHSs had many components, and many included data from either patient surveys or patient charts. The processes varied widely, from engaging patients in the process to exclusively analyzing the data. The outcomes were largely patient-reported, though several clinical outcomes were also used to benchmark the success of the LHS.

Discussion

Our review shows that LHS definitions, structures, processes, and outcomes in 3C applications vary widely. Many have shown substantial potential to be implemented and improve care in 3C populations. To deliver on this goal, future work will need to focus on better specification, formalization, and definition of LHS approaches, as well as better design of their structures, processes, and outcomes to fit the needs of the intended population.

Keywords:

learning health system
chronic health conditions
quality improvement
implementation research
feedback data
feed forward data
patientcentered registry
==== Body
pmcIntroduction

A learning health system (LHS) is a systems model where data from day-to-day health care encounters are utilized to create a feedback loop for continuous improvement.1 LHS components have been implemented around the world since the formal inception of the LHS model in 2007 by the Institute of Medicine.2 The Agency for Healthcare Quality and Research (AHRQ) criteria for LHS inclusions defines a LHS as “a health system in which internal data and experience are systematically integrated with external evidence, and that knowledge is put into practice,” with the end result being that patients receive better, safer, more efficient health care.3 The 2001 landmark report, “Crossing the Quality Chasm,” highlighted to the American medical community that US health care was grossly inefficient and much less effective than would be expected for a highly industrialized nation, particularly at incorporating medical research findings and innovation.4 A proposed solution was the LHS. There has been a proliferation of LHS applications and related concepts, including learning health networks and learning health organizations, and their applications are varied across health care concepts. The present review focuses on applying the LHS concept fitting AHRQ criteria, allowing for variation in contexts (e.g. health systems, multicenter applications, single-center studies).

Ninety percent of the annual health care expenditures in the US are for people with chronic and mental health conditions.5 These complex, chronic, and costly (3C) conditions cause extensive strain on patients and their families and are challenging to manage by practitioners. LHSs have been applied in a wide variety of such conditions, and there has already been some success in LHSs being applied to 3C conditions in both adult and pediatric diseases. In pediatrics, the PEDSnet national LHS, founded in 2009, tackled chronic pediatric conditions, often rare diseases, and reshaped their care, redefining outcomes for previously fatal diseases, such as cystic fibrosis.6 In adults, ImproveCareNow, a learning health network, has consistently improved inflammatory bowel disease (IBD) care.7

In this scoping review, the authors sought to review the established literature to describe and synthesize the current state of the art and science of LHSs as applied to 3C health conditions. LHSs have been evaluated in several reviews; however, in the present review, the authors wanted to focus on some of the most logistically and medically challenging conditions and observe how a LHS approach can ameliorate some common roadblocks. This review was conducted with the purpose of informing key stakeholders on how best to implement a LHS for a new post-acute COVID syndrome LHS in development in a rural New England academic health system.

Methods

Review structure

The authors structured the present review on the basis on the Donabedian framework for evaluating the quality of health care. Originally developed in 1966 by Avedis Donabedian, the Donabedian model has stood as a standard for evaluating the quality of health services. The model frames the quality of health care within 3 simple components: 1) structure, 2) process, and (3) outcomes. “Structure” is defined as the environment in which care is being provided, such as the physical resources (the space, the amenities, the money), the human resources, and the way the setting is organized and operates.8 “Process” includes the activities of the patient, both in searching for and following through with care, as well as the actions of the caregiver, encompassing diagnosis and treatment. The third component, “outcome,” refers to all effects of the care on the individual patient or collective.

The authors used the Donabedian model as an effective lens through which to evaluate very diverse LHSs, with some framework to maintain cohesiveness. This unique approach is both a strength and a weakness. Evaluating LHSs this way can illuminate distinctions; however, this is a different approach than taken by other reviews. The components of the Donabedian model align well with the AHRQ’s 6 LHS criteria, which include the following: 1) leaders committed to learning and improvement, 2) systematically gathering and applying evidence, 3) using information technology (IT) methods to inform shared decision-making, 4) promoting inclusion of patients as vital members of the learning team, 5) capturing and analyzing data to improve care, and 6) continually assessing outcomes to inform improvement and research.5 The structure component of the Donabedian model aligns with but is not limited to the following AHRQ LHS criteria: element #1 (leaders), #3 [IT (infrastructure)], and #4 [inclusion of patients (personnel)]. The process component in the Donabedian model applies to but is not limited to all 6 elements in terms of the actions a LHS performs. Finally, the outcomes component aligns with but is not limited to AHRQ LHS criteria element #6 (outcomes for improvement and research).

Search strategy

We adhered to the 5-stage methodological framework for scoping studies by Arksey and O’Malley,9 as well as the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR; see Appendix 3).10 The authors’ search strategy, designed with the help of a biomedical librarian, aimed to locate both published and unpublished studies. An initial limited search of Medline was undertaken to identify articles on the topic. The text words contained in the titles and abstracts of relevant articles, as well as the index terms used to describe the articles, were used to develop a full search strategy for Medline, Cumulative Index to Nursing and Allied Health Literature, and Scopus (see Appendix 1). The protocol was registered with the Open Science Framework platform (OSF identification no. osi.io/7yzup) The search strategy, including all identified keywords and index terms, was adapted for each included database and/or information source. The reference list of all included sources of evidence was screened for additional studies. Only studies published in the English language were included. Sources of unpublished studies/gray literature were searched, including in Google Scholar. Studies published since 2007 were included, as that is when the concept of LHS was introduced at a workshop organized by the Institute of Medicine.2 Study authors were contacted if data were absent or if additional clarification was needed. Identified citations were uploaded to EndNote 20 to identify and remove duplicates. Next, citations were inputted into Rayyan software for title/abstract screening, followed by full-text screening.11 Two authors conducted the title/abstract screening (R.A.D. and K.S.).

Selected studies

To be included, an article had to 1) evaluate a LHS that had been piloted, 2) focus on a chronic disease, and 3) recognize part of the Donabedian model’s structure–process–outcome quality-of-care model. Two authors conducted the full-text screening (R.A.D. and K.S.). To perform an in-depth and broad research, no article was excluded based on type or place of publication. However, the authors screened out articles that were simply project proposals or small-scale pilot programs. Articles were excluded if 1) they did not focus on a LHS, 2) the implemented LHS did not specifically focus on targeting chronic conditions, 3) they did not have both feedforward and feedback components, or 4) they were not focused on technology aspects enabling data collection, registries, or other LHS characteristics. Several articles described tools or other prototypes that would become components of a LHS but that in isolation did not meet the criteria for inclusion. The initial screening process was followed by a full-text screening using the previously established criteria. Reasons for exclusion were recorded and reported in accordance with PRISMA-ScR standards.9 Discrepancies between reviewers were discussed to obtain a consensus. The authors did not formally assess the chance of bias of included studies or outcomes. This review did not meet the definition for human subject research and did not need institutional review board approval. Finally, in the review’s subsequent discussion of the included studies, the authors acknowledge 2 key aspects: 1) there is substantial variation across studies in terms of approach and level of detail reported, and 2) that the authors were not able to report all relevant details on all included studies (rather, the review’s focus was on key characteristics and exemplars to emphasize).

Data extraction

Data were extracted from papers included in the scoping review by 2 independent reviewers (R.A.D. and E.B.C.) using a data extraction tool developed by the reviewers (see Appendix 2). The data extracted included specific details about the participants, concept, context, study methods, and key findings relevant to the review question. Due to the wide range of interventions, study designs, and outcomes, a narrative synthesis method was used. The authors based the data collection for this review on the Donabedian model for evaluating health care improvement.12 Information was organized on the data charts and was used to assemble and chronicle the papers’ approaches toward implementing a LHS for a 3C condition.

Results

Study characteristics

The authors’ search identified 656 titles after duplicate removal, including 2 articles returned after a gray literature search and reference review. Screening of titles and abstracts excluded 600 of these due to failure to meet full inclusion criteria. The remaining 56 full-text articles were examined and a further 39 were excluded for not meeting the full criteria. This left 17 articles meeting the full inclusion criteria. Search results are summarized in the PRISMA flow diagram (Figure 1). Of the 17 studies included, 6 focused on cancer, 4 on mental health, and 3 on IBD (Table 1). The remaining articles focused on type 1 diabetes (n = 1), pain (n = 1), and general chronic illnesses (n = 2). All these LHS were conducted in the US, except for 1 study that took place in Canada.10

Figure 1: LHS PRISMA Flow diagram. Flowchart created using PRISMA design from Page MJ, McKenzie JE, Bossuyt PM et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.13 DOI: 10.1136/bmj.n71. 36 Creative Commons Attribution (CC BY 4.0) license (https://creativecommons.org/licenses/by/4.0/legalcode).

Table 1: Characteristics of included articles

Author(s), year	Article title	Journal	Institution, country	Population	Program name	Purpose of study	Structure	Process	Outcome(s)	Results	
Abernethy et al, 201014	Electronic patient-reported data capture as a foundation of rapid learning cancer care	Medical Care	Duke University, USA	Cancer, breast, and gastrointestinal	NA	"To show how rapid learning health care might be implemented and structured, function in clinical practice, and support comparative effectiveness research"	Patient-reported symptom data collected with tablet computers	Problem was identified and subsequently characterized using data; clinical staff came together to develop an intervention, which was then implemented and evaluated	Electronic PROs, psychosocial	Electronic PRO system is feasible	
Almario et al, 202115	Health economic impact of a multicenter quality-of-care initiative for reducing unplanned health care utilization among patients with inflammatory bowel disease	American Journal of Gastroenterology	Karsh Division of Gastroenterology and Hepatology, Cedars-Sinai Medical Center, USA	IBD	IBD Qorus	To estimate "the health economic impact of participation in the Collaborative for IBD Qorus, as a whole, and for each individual site"	Only site-level, de-identified data was used from the participating institutions. Overall, 23 (13 academics and 10 communities) of 27 sites provided data to support clinical probability estimates	Investigators used decision analysis software to model the economic impact of participation in the IBD Qorus BTS Collaborative	Reducing unplanned health care utilization	100% of the simulations showed that the interventions were cost saving vs the baseline period when considering all sites collectively	
Azar et al, 201516	The Indiana University Center for Healthcare Innovation and Implementation Science: Bridging health care research and delivery to build a learning health care system	Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen	The Indiana University Center for Healthcare Innovation and Implementation of Science (IU-CHIIS), USA	Dementia and depression	NA	"To use implementation science and innovation to produce great-quality, patient-centered and cost-efficient health care delivery solutions for the United States of America"	Data monitoring	Strategies included a network of leaders who support the exchange of information among partner health care systems, rapid cycle research and discovery unit in each health care systems partner, developing, evaluating, implementing, and disseminating innovative health care solutions and intellectual properties of novel discovered solutions, and training and mentoring health care practitioners, researchers, and health system managers and leaders	Implementation science and innovation to produce great-quality, patient-centered and cost-efficient health care delivery solutions for the US	Scaled up an evidence-based dementia and depression collaborative care model; expanded the accountable care unit leading to reduction in length of stay, readmissions, mortality and central line associated blood stream infection rate; created first certificate in innovation and implementation science in US and secured funding	
Bauer et al, 201917	The collaborative Chronic Care Model for mental health conditions	Medical Care	Veterans Health Administration	Mental health	Chronic Care Model (CCM)	To "describe a multistage effort that evolved over 8 years to move the CCM into broad usage in the VA for mental health conditions"	Health system/researcher partnership	Monthly meetings to discuss any issues implementing the model	Clinical outcomes (eg, number of diagnoses)	Evidence produced leverage to create policy changes and the VA’s Office of Mental Health and Suicide Prevention adopted the CCM	
Bhandari et al, 201618	Pediatric-Collaborative Health Outcomes Information Registry (Peds-CHOIR): A learning health system to guide pediatric pain research and treatment	Pain	Stanford University, USA	Pediatric pain	Peds-CHOIR	"To describe the first application of CHOIR in a pediatric pain clinic (Peds-CHOIR), with emphasis on the dual tracking capacity for patient and caregiver reported outcomes"	Has capabilities to capture data at each clinic visit; display graphical, real-time results that inform point-of-care decisions; and track patient treatment responses longitudinally	Registry emphasizes tracking of patient-generated information as a core component of clinical practice, allowing for individualized improvements in the health care delivery process over time and guiding precision pain medicine	Pain management improvements at point-of-care	Data inform assessments, patient and caregiver education needs, and individual- and family-based interventions specific to clinical presentation and patient preferences; graphical presentation of progress allows for interactive involvement of the patient and caregiver with the clinician in informing treatment recommendations at point-of-care	
Bozkurt et al, 202019	Phenotyping severity of patient-centered outcomes using clinical notes: A prostate cancer use case	Learning health systems	Stanford University, USA	Cancer, prostate	NA	To "propose a natural language processing pipeline to identify the severity of urinary incontinence in prostate cancer patients using only free-text clinical notes from EHRs"	Run EHR clinical notes through program	Use an NLP software to create rule-based phenotyping from clinical notes which contain patient-centered outcomes from a previously developed dictionary	Correct identification of urinary incontinence into categories	The rule-based model correctly identified patients into predetermined categories of severity as well as outperformed the deep-learning model	
Fung-Kee-Fung et al, 201810	Regional process redesign of lung cancer care: a learning health system pilot project	Current Oncology	Ottawa Hospital, Canada	Cancer, lung	Ottawa Health Transformation Model	To demonstrate "the application of the Ottawa Health Transformation Model to create effective redesign of the regional system for lung cancer diagnostic services"	Input from clinicians, administrations, leaders, and patients; automated, sustainable technology	Central intake, joint review by multidisciplinary team, concurrent consults with multiple practitioners, navigation day for multiple appointments, dashboard for consults, standardized processes for result review, testing wait times, automated workflows	Total duration of the patient journey from referral to receiving a cancer treatment	Decreased median patient journey by almost half	
Johnson et al, 201720	Fostering collaboration through creation of an IBD learning health system	American Journal of Gastroenterology	The Dartmouth Institute of Health Policy and Clinical Practice	IBD	IBD Qorus	Describes the "describes the overarching framework of a learning health system, the application of the framework to IBD Qorus— including the patient-physician relationship at the center that fosters co production of care—and quality improvement efforts that IBD Qorus has initiated using this approach"	Combines patient-reported and clinical data into one view and is accessible to both patient and physician. The patient reports much of the data that is captured on the dashboard using a web-based pre-visit health assessment	Created visual dashboard	 "Advance evidence-based care and improved outcomes for IBD patients "	"Optimal health and high value care for adults with Crohn’s and colitis"	
Kamal et al, 201221	Improving the management of dyspnea in the community using rapid learning approaches	Chronic Respiratory Disease	Duke University, USA	Dyspnea	Duke Cancer Care Research Program	"Results of important studies in the management of chronic respiratory disease are presented in brief; however, the focus of this review is on evidence supporting the implementation of a rapid learning model for symptom management"	Electronic user interfaces (e.g. tablet computers) and digital pens to collect electronic PROs at the point-of-care. In 2.0 patients with cancer complete an 80-item (86 for women) review of symptoms instrument while in the waiting room	Data are securely transferred to a central server to be aggregated with clinical results and maintained for longitudinal storage and analysis	Patient-reported	Concluded that recent findings suggest that a rapid learning system is feasible and acceptable to patients with advanced illness, helps monitor symptoms overtime, facilitates study of the impact of novel interventions, and can identify unrecognized needs and concerns	
Kilbourne et al, 202122	Learning health systems: Driving real-world impact in mental health and substance use disorder research	Federation of American Societies for Experimental Biology Bioadvances	Veterans Administration, USA	Mental health and substance use disorders	NA	Using LHS framework, to describe "describe current Veterans Affairs research initiatives in mental health and substance use disorders that rigorously evaluate national programs and policies designed to reduce the risk of suicide and opioid use disorder (data to knowledge); test implementation strategies to improve the spread of effective programs for Veterans at risk of suicide or opioid use disorder (knowledge to performance); and identify novel research directions in suicide prevention and opioid/pain treatments emanating from implementation and quality improvement research (performance to data)"	EHR data	Include implementation of quality improvement strategies and rapid-cycle evaluation methods to test and validate interventions in real-world care delivery settings	Point to several insights into building an LHS to support ongoing translation of research into practice	Assessments of these LHS-inspired initiatives are still in process, limitations in the authors’ review include the lack of complete information on the LHS impacts, notably on cost and quality of care over time	
Marsolo et al, 201523	A digital architecture for a network-based learning health system: Integrating chronic care management, quality improvement, and research	Generating Evidence & Methods to Improve Patient Outcomes (EGEMS)	Multiple	Chronic conditions	ImproveCareNow	"To create a proof-of-concept architecture for a network-based LHS"	Data capture and input (using web forms, electronic data management, and consent status) to be used for analytics (using a registry, data sets, data quality measures, population measures, quality improvement measures, and medical results)	Creation of automated reports	10 outcome measures (eg, percentage of patients in remission), 7 process measures (eg, percentage of visits where initial dose of anti–tumor necrosis factor therapy is given and patient had a tuberculosis test within the prior 12 months), and 8 data quality measures (eg, percentage of hospitalizations entered within 30 d of discharge)	Reports are highly utilized and uptake of the system is growing	
Mayo et al, 201724	Qualitative study of oncologists’ views on the CancerLinQ rapid learning system	Journal of Oncology Practice	University of Michigan, USA	Cancer	CancerLinQ	"To explore providers’ opinions and concerns related to implementation of CLQ, including ethical issues"	see Potter et al25	See Potter et al	See Potter et al	Support for usage of big data, but concern over patient privacy and whether participation should be optional	
Potter et al, 202025	Development of CancerLinQ, a health information learning platform from multiple electronic health record systems to support improved quality of care	JCO Clinical Cancer Informatics	63 organizations across the country, USA	Cancer	CancerLinQ	To "describe the process by which we ingest EHR data"	EHR data extraction, dashboard creation	Assists practitioners and patients with decision-making	Clinical	Looks promising to help get more patients enrolled in clinical trials	
Rubinstein and Warner, 201826	CancerLinQ: origins, implementation, and future directions	JCO Clinical Cancer Informatics	Vanderbilt, USA	Cancer	CancerLinQ	To "review the impetus behind and inception of American Society of Clinical Oncology’s CancerLinQ (Cancer Learning Intelligence Network for Quality) initiative"	See Potter et al	See Potter et al	See Potter et al	NA	
van Deen et al, 202127	The reliability of patient self-reported utilization in an inflammatory bowel diseases learning health system	Crohn’s & Colitis 360	Cedars-Sinai Medical Center, USA	IBD	IBD Qorus	To determine "reliability of patient self-reported utilization in an inflammatory bowel diseases learning health system"	Data extracted from the EHR and patient self-reports within the IBD Qorus LHS	Comparing the electronic records to patient self-reports	EHR encounters	IBD patients’ self-report of hospital utilization and medication use is highly accurate, with agreements between self-report and the EHR of ≥89% for all utilization and medication measures	
Weinstock et al, 202128	T1D Exchange Quality Improvement Collaborative: A learning health system to improve outcomes for all people with type 1 diabetes	Clinical Diabetes	SUNY Upstate Medical University, Syracuse, USA	Type 1 diabetes	T1DX-QI	Presents the results of 2 of the clinical interventions undertaken by the T1D Exchange Quality Improvement Collaborative (T1DX-QI)	Data from electronic health records	Improvement collaborative and electronic health record multicenter collection and comparison	 "Address inequities in type 1 diabetes care"	Conclusion is that use data to drive continuous quality improvement, share best practices, overcome therapeutic inertia, confront racism, address SDOH, and advocate to accelerate change and address the needs of all people with diabetes	
Zandi et al, 202029	Development of the National Network of Depression Centers Mood Outcomes Program: A multisite platform for measurement-based care	Psychiatric Services	The National Network of Depression Centers is a nonprofit consortium of 26 leading clinical and academic member centers in the US	Depression	NA	"The NNDC has established a measurement-based care program called the Mood Outcomes Program whereby participating sites follow a standard protocol to electronically collect patient-reported outcome assessments on depression, anxiety, and suicidal ideation in routine clinical care. This article describes the approaches taken to develop and implement the program"	Electronically collected PRO assessments on depression, anxiety, and suicidal ideation in routine clinical care	"Data are uploaded to a secure repository and a summary report is generated for the clinician"	characterize symptom burden in a disease-specific manner (and ultimately, in a treatment-specific manner) so that interventions can be designed and tested	Demonstrates the potential of the Mood Outcomes Program to create a nationwide LHS for mood disorders	
BTS, breakthrough series; CLQ, CancerLinQ; EHR, electronic health record; IBD, inflammatory bowel disease; LHS, learning health system; NA, not applicable; NLP, natural language processing; NNDC, National Network of Depression Centers; PROs, patient-reported outcomes; SDOH, social determinants of health; SUNY, State University of New York; VA, Veterans Affairs.

Included LHSs

The studies that focused on cancer varied in cancer pathology and LHS characteristics. Duke University’s LHS was designed to test the feasibility of using electronic patient-reported outcomes (PROs) in breast and gastrointestinal cancer patients for use in a rapid learning–based LHS.21 The electronic PRO data were collected using tablets and used in real time to support comparative effectiveness research.14 Stanford University designed a LHS for patients with prostate cancer that used natural language processing (NLP) software to classify severity of their disease to increase availability to precision-based therapy.19 At the Ottawa Hospital, a LHS pilot project was initiated to create a redesign of the regional system, and input was taken from their multidisciplinary health care team to better coordinate regional treatment for lung cancer.10 The American Cancer Society developed CancerLinQ, a rapid learning system that uses electronic health record (EHR) data in real time for quality improvement and research. The platform takes unstructured data from patients and ingests it into their database for use in increasing data from patients not enrolled in clinical trials.25 CancerLinQ works in the background of clinical systems, collecting both long- and short-form data.26 When oncologists were surveyed about their views about CancerLinQ, it was found that, although the physicians supported the use of big data in cancer care, they were worried about patient privacy.24

Four LHSs focusing on mental health were included. 1) The Indiana University Center for Healthcare Innovation and Implementation Science scaled up an evidence-based dementia and depression collaborative care model to produce great-quality, cost-efficient, and patient-centered health care.16 2) The Chronic Care Model at the Veterans Health Administration used a LHS system to improve patients’ mental health clinical outcomes. This multicomponent model included linkage to community resources, practitioner decision support, patient self-management support, and use of clinical information systems.17 3) The Veterans Health Administration used the LHS model to help patients with mental health challenges and substance use disorders. This included Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment, a suicide prevention program that identified high-risk veterans, and Stratification Tool for Opioid Risk Management, which used a real-time data dashboard to present patient risk and work to mitigate it.22 4) The National Network of Depression Centers created the Mood Outcomes Program, which collected electronic PROs with the goal of establishing a nationwide LHS for mood disorders to improve patient outcomes.29

Studies focusing on IBD included the IBD Qoros and ImproveCare Now systems. The IBD Qorus Collaborative was designed to decrease unplanned emergency department trips and hospital admissions for patients with IBD at both academic and community medical sites. IBD Qorus relies on a coproduction model that includes collaboration between patients and their care teams using feedforward data and a shared data platform.20 Patient self-reports were found to be reliable.27 It was also reported that participating in the collaborative was associated with lower expenditures per patient.15 The Cincinnati Children’s Hospital Medical Center worked with ImproveCareNow, a LHS registry for children with chronic gastrointestinal diagnoses, to create an EHR system that produces LHS analytic reports in order to improve a variety of outcomes and assist with comparative effectiveness research.23

Other studies to highlight include the Pediatric-Collaborative Health Outcomes Information Registry (PEDS-CHOIR) and the T1D Exchange Quality Improvement (T1DX-QI) collaborative. PEDS-CHOIR, created by Stanford University, was designed to serve as a free, flexible, open-source LHS to improve pain management at the point-of-care.18 The web-based interface tracks patient-generated information and uses these data to inform interventions. The T1DX-QI collaborative uses EHR data to address inequities in type 1 diabetes treatment and piloted quality improvement interventions, such as increasing the use of insulin pumps.28

LHS structures

The structures of the studied LHSs relied on registries, usually using EHRs, or PROs. They ranged from real-time dashboards to inform patient care to more longitudinal probability estimates. CancerLinQ, one of the more established LHSs in the present review, utilized both individual patient data, to help oncologists make recommendations, and population-level data, to improve cancer treatment and contribute to research.

Processes of LHSs

The majority studies used feedforward data (14, 82.4%) and feedback data (13, 76.5%) (Table 2). In some cases, patients filled out a form via a tablet, and the data were subsequently used to design health care delivery. Other times, data were captured via charts, claims, or clinical notes to inform the feedforward and feedback processes. There were also some scenarios where physicians, other health care practitioners, or patients were involved in the LHS processes. One notable example of utilizing user feedback is the Ottawa Health Transformation Model, which made patient input a key feature of their codesign and coproduction process.10

Table 2: Components of included articles

Author(s), year or program name	Feed forward data	Feedback data	Registry enabled	Improvement collaborative	Implementation science	Innovation incubator	Research	PROs data	EHR data	Claims data	Qualitative data	Codesign	Coproduction	Peer facilitated network	Multicenter	Visualization (dashboards)	
Abernethy et al, 201014	✓	✓	✓		✓			✓	✓								
Almario et al, 202115	✓	✓			✓		✓	✓			✓		✓		✓		
Azar et al, 201516	✓	✓		✓	✓	✓	✓	✓			✓	✓	✓	✓		✓	
Bauer et al, 201917				✓	✓										✓		
Bhandari et al, 201618	✓	✓	✓		✓		✓	✓			✓				✓		
Bozkurt et al, 202019	✓		✓				✓	✓	✓		✓						
CancerLinQ: Potter et al,25 Mayo et al,24 Rubinstein and Warner,26	✓	✓	✓						✓						✓	✓	
Fung-Kee-Fung et al, 201810	✓	✓			✓			✓	✓		✓	✓	✓	✓			
Johnson et al, 201720	✓	✓			✓			✓				✓	✓		✓	✓	
Kamal et al, 201221	✓	✓		✓	✓		✓	✓			✓	✓	✓	✓			
Kilbourne et al, 202122	✓	✓			✓		✓		✓	✓		✓	✓		✓		
Marsolo et al, 201523	✓	✓	✓	✓	✓				✓						✓	✓	
van Deen et al, 202027							✓	✓	✓								
Weinstock et al, 202128	✓	✓		✓	✓		✓		✓			✓	✓	✓	✓		
Zandi et al, 202029								✓							✓		
EHR, electronic health record; PROs, patient-reported outcomes.

Outcomes

Over half of the studies, 11, used PROs to guide their LHS. Eight studies used EHR data. The authors looked at quantitative outcomes, such as percentage of patients in remission, as well as qualitative outcomes, such as health care practitioner opinions of a tool. Several papers noted the importance of repeated measurement of multiple outcomes in order to fairly evaluate their program. The T1DX-QI collaborative was the only LHS included that focused specifically on the social determinants of health and how they relate to outcomes.28

The LHSs did not report results uniformly, but all presented some form of positive outcome (Table 1). Many papers showed the proof of feasibility of LHSs and highlighted opportunities for growth and additional usages.14,18,25,29 Others measured outcomes directly, by showing reduction in mortality,16 cost savings,15 and improvement in patient experience.10 Some LHSs were able to provide date output that would affect medical care, such as enrollment in clinical trials or categorization of illness.19,25 Still others focused on the public health potential of LHSs, including informing policy17 and addressing social determinants of health.28

Challenges of LHSs

Many of the LHSs piloted encountered a variety of challenges. Many noted that setting up their LHS took a large amount of initial time and investment. It was a challenge to acquire stakeholder alignment and have the organization commit to the LHS.17,23 Feedback is essential to a LHS and a lack of follow-up survey completion was an issue noted in some studies.22 Other studies noted that a major challenge was acquiring both the funds and personnel required to establish and sustain the LHS.20 The design of a good IT system that supports both clinical and quality improvement uses can be time consuming and costly.17 There were also technological limitations, mainly around obtaining data accurately and completely from clinical narratives.25,27 Among multicenter LHSs, heterogeneous data elements, legal issues, and competition created problems when developing and expanding a LHS.26 Expanding LHSs resulted in the challenge of working with varying EHR vendors and IT systems,14,26 as well as differences in LHS assessment and implementation across teams.15,17

Exclusions with LHS potential

There were many noteworthy studies with LHS potential the authors excluded while conducting the present review. For example, Melanoma Rapid Learning is a clinical decision support system based on melanoma cohorts that includes some components of a LHS, providing feedback, but lacking feedforward components.30 Multicenter databases that make vital information available to patients and physicians with the purpose of improving quality of care show great promise but do not yet document inclusion of all elements of a LHS. These multicenter initiatives include the Multiple Sclerosis Continuous Quality Improvement Collaborative and the Dartmouth Long COVID Learning Health System.31 Finally, machine learning and NLP are also emerging as new characteristics of the LHS effort. A recent prostate cancer study utilized NLP tasks to analyze unstructured clinical narratives in the EHR to classify disease severity. These NLP tasks created new patient outcomes data that could be incorporated in a LHS.23 Although advancements in machine learning were outside the scope of this review, recent proposals call for the integration of machine learning data within LHSs to advance precision medicine radiation oncology treatments.32 The Multiple Sclerosis Continuous Quality Improvement Collaborative has recently explored predictive analytics models for predicting multiple sclerosis exacerbations (also known as relapses).33

Plans for additional LHSs for chronic conditions

Plans for additional LHSs hold impressive promise for furthering the care of 3C conditions. In a 2014 report, the Institute of Medicine’s Committee on Improving Quality Cancer Care highlighted advancing learning health care IT as essential to improving cancer care.31 Examples of oncology projects maturing to LHS potential include the standardized inter-center data collection Breast Large Database project, applying the World Health Organization’s implementation methodology Integrative Systems Praxis for Implementation Research to eliminate cervical cancer and a LHS pilot to support smoking cessation interventions in cancer centers.33–36 The National Institute on Aging’s Imbedded Alzheimer’s Disease and Related Dementias Clinical Trials Collaboratory proposes a LHS model to transform dementia care by learning continuously from clinical trials.37,38

Discussion

Principal findings: Variation in LHS efforts in 3C conditions

In this scoping review, the authors have endeavored to describe LHSs that focused on improving and studying care in 3C populations and found substantial variation in LHS characteristics, design, function, scope, and purpose, suggesting a very early stage of overall development and application overall of LHS efforts in 3C populations. Few were rigorously designed for research or implementation using a formal implementation framework. Despite this, many LHS efforts made progress toward their aims and improved outcomes even in the absence of complete or perfect LHS design and functioning. Throughout the present review of LHS, many ethical questions were revealed, such as how much patient data should be shared and who should have access to this data. This presents an area of growth as the number of LHS grows and their maturity develops. This is consistent with the work by Ellis et al., which addressed the scope, focus, and spread limitations of current LHS efforts.39

Challenges faced by 3C LHS efforts

The use of the term LHS can mean many things. There are many varying definitions of a LHS and many different purposes for them, ranging from very specifically defined research protocols to improvement learning collaboratives informed by LHS data structures.3 Many of the LHS applications in the present review were oriented toward improvement, and others toward research and/or implementation. The variability in LHS design is clearly illustrated in Table 2 of this scoping review. In this review, the authors noted studies that were notable exclusions with LHS potential or initiatives promising to meet the authors’ full LHS definition in the future. It is also challenged that LHS can be confused with overlapping terms, such as learning health networks and communities of practice, which are related but not exactly the same.40,41

CancerlinQ, the most evolved LHS in the present review, which was established in 2015, has encountered numerous barriers in implementation, such as improving data interoperability and utility, mitigating legal and user trust issues, and remaining competitive in the environment of emerging LHSs, as well as achieving the satisfaction of multiple stakeholders, such as patients, practitioners, researchers, and administrators. Other concerns included patient privacy, data sharing, and patient consent. Maintaining actionable linkages between the conceptual model a LHS is built upon, its actual design (structure), and how it functions in practice (process) is not always attained or sustained.

A preponderance of LHS work to date has focused on the technical design and process aspects, often to the detriment of underemphasizing the importance of human factors and the contributions of people with 3C conditions to the overall success of the LHS itself.42 Work in the coproduction field has emphasized the critical importance of this.43,44 Finally, the evidence base for LHS evaluation is very early in its development. To date, very few LHS efforts, including those regarding 3C conditions, have been evaluated using formal frameworks or rigorous methods,39 including in 3C conditions, and there are many challenges in assessing improvement in general and in 3C populations.13 This creates a challenge in assessing if LHS structures and processes used in each initiative are contributing to (or impeding) the reported outcomes. This also raises the question if the LHS approach being utilized is appropriate for the 3C population and intervention being studied. Furthermore, evidence-based recommendations have not yet been established for when a LHS approach may be indicated (or not) for a particular 3C intervention study or implementation or improvement effort. Given the high cost and substantial effort required to establish and maintain LHS efforts, more work is needed in the evaluation of LHS efforts.

Ethics and equity

It has been argued that it is an ethical responsibility for patients to contribute to improved future care in a system in which they may benefit.39 Additionally, scientific citizenship demands patient engagement and understanding of how their data could be used to generate knowledge.45 Three themes in particular have emerged: 1) informed consent, 2) trust, and 3) acceptable data uses. According to the Health Insurance Portability and Accountability Act and the Federal Policy for the Protection of Human Subjects (the Common Rule), the opt-out system for informed consent is legal.10,46 Focus groups enrolling 217 CancerLinQ patients found that although patient preference varied, the majority (54.9%) of patients chose an opt-out policy as an appropriate ethical balance between what is best for society and the individual.10,12 However, in a nationwide, racially diverse survey of 621 CancerLinQ patients, the majority of patients recognized the importance of secondary users of data (72%) but also most endorsed an opt-in consenting process at least once (71%).45 Black and Hispanic patients were more likely to indicate preference for opt-in consenting each time data would be used.42 This is likely due to preexisting medical mistrust. Patients rating the highest levels of trust in the medical system tended to be more accepting of the less formal consent process for LHS.41 Patient trust in the medical system was also tied to comfort with de-identified data use. Trust of end users was particularly important.41–43 Patient perceptions of appropriateness of LHS were highest when the data would be used by physicians or university researchers.14,42

Recent research demonstrates patients’ distrust of pharmaceutical and insurance companies, especially regarding pharmaceutical and insurance companies’ involvement with an LHS in which data is used for marketing purposes.39,40,43,45,47 Finally, patients desire transparency and access to their data.47 Transparency plays a vital role in the consenting processes, such as distributing easily understandable LHS information materials and clearly delineating opt-in or opt-out processes.39,47 Given these concerns, messaging about LHS efforts must be transparent, clear, effective, and context specific to the 3C population involved with the LHS concerning the involvement of stakeholders, as well as the purpose, design (structure), methods (processes), and goals (outcomes). Great care must be taken not to leave the people with 3C conditions the researchers are trying to study and help behind in the pursuit of progress.13

Strengths and limitations of the present study

The present review included articles from across the US and Canada, with studies ranging from 178 to 1,426,015 patients. The authors’ search strategy was advised by a professional biomedical research librarian and included a wide range of sources and article types. Despite the close collaboration with the librarian, those working in LHSs and the authors’ research team, not all 3C conditions were explicitly stated in the search strategy for this review. The authors believe that the strategy that was developed was strong enough to capture LHSs focusing on 3C conditions that were beyond the specific ones identified with key words. Due to the nature of the scoping review method, the authors did not appraise the quality of evidence of the articles included. This could result in the inclusion of low-quality evidence in the present analysis. In addition, the authors may have missed some eligible studies during the review, though the chance of this was decreased due to the use of duplicate screeners and the use of a structured process. The authors’ use of the Donabedian model provides both the potential benefit of introducing a new frame of describing included LHS examples in terms of structure, process, and outcomes. This strength can also be a potential risk that this may prove to be a suboptimal lens for this project. The authors did not include some studies written in languages other than English and those that were early stage ideas, such as proposals or illustrative case studies. This may have excluded important new innovations developing in the field at the time of this review and which may have future promise. In analyzing the studies, the authors noticed a wide variety of approaches with a myriad of intricacies. Due to space, the authors were unable to emphasize all articles in depth and, therefore, selected some exemplars to describe in detail for each section.

Future implications

Many initiatives have been attempted in 3C populations in improvement, implementation, and/or research applications—some of them with an LHS approach. LHS definitions, structures, processes, and outcomes in 3C applications currently vary widely, but many have shown progress and future potential and may develop into effective and efficient means to simultaneously provide care, study care, and improve care in 3C populations. However, to deliver on the aspirational goals to which LHSs aspire, future work will need to focus on better formalization and specification of LHS approaches, such as better design of structures, processes, and outcomes to fit the needs of the people whose care is included in an LHS. If this can be achieved and maintained in an ethically balanced manner, then acceleration in research, translation, improvement, dissemination, and implementation may be achieved for the greater good of all.

Supplementary Material

Appendix 3

Appendix 1

10.7812/TPP/24.012.supp3 Appendix 2

Acknowledgments

The authors thank Ms Heather Blunt from the Dartmouth Biomedical Library for her assistance with the search strategy and Drs Shani Bardach and Jeffrey Parsonnete for thoughtful reviews of this manuscript prior to submission. Ms Davis was supervised by Dr Oliver as a MPH student intern as part of the requirements of her MPH program at The Dartmouth Institute (completed May 2022), during which time she completed substantial supervised work leading the initial development of this manuscript.

Author Contributions: Study concept and design by Rebekah A Davis, MPH, Brant J Oliver, PhD, MS, MPH, FNP-BC, PMHNP-BC, and Jabeen Ahmad, PhD, MPH. Data collection by Rebekah A Davis, MPH, and Kathryn Sine. Data analysis by Rebekah A Davis, MPH, and Debra Ella Burguera-Couce. Manuscript preparation by Rebekah A Davis, MPH, Kathryn Sine, Debra Ella Burguera-Couce, and Jabeen Ahmad, PhD, MPH. All authors participated in interpretation of results and critical revision of the manuscript for important intellectual content.

Conflicting of Interests: None declared

Funding: This study was supported in part by The Dartmouth Institute Susan and Richard Levy Health Care Delivery Incubator (https://sites.dartmouth.edu/levyincubator/) and the MPH program at The Dartmouth Institute for Health Policy and Clinical Practice. The funder was not involved in the design, data collection, interpretation, or publication of the scoping review.
==== Refs
References

1. Bioethics J . What is a Learning Health System? Accessed 6 July 2024. https://bioethics.jhu.edu/learning-health-systems/about
2. Olsen LA , McGinnis JM , eds. Institute of Medicine (US) Roundtable on Evidence-Based Medicine. In: The Learning Healthcare System: Workshop Summary. Washington DC: National Academies Press (US); 2007.
3. AHRQ . About Learning Health Systems. 2019. Accessed https://www.ahrq.gov/learning-health-systems/about.html
4. IOM . Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care: Workshop Series Summary. Grossmann C , Powers B , McGinnis JM , eds. National Academies Press(US); 2011.
5. Centers for Disease Control and Prevention (CDC) . Health and Economic Costs of Chronic Diseases 2022. Accessed 24 July 2024. https://www.cdc.gov/chronic-disease/data-research/facts-stats/?CDC_AAref_Val=https://www.cdc.gov/chronicdisease/about/costs/index.htm
6. Forrest CB , Margolis PA , Bailey LC , et al. PEDSnet: A national pediatric learning health system. J Am Med Inform Assoc. 2014;21 (4 ):602–606. 10.1136/amiajnl-2014-002743 24821737
7. Seid M , Hartley DM , Dellal G , Myers S , Margolis PA . Organizing for collaboration: An actor-oriented architecture in ImproveCareNow. Learn Health Syst. 2020;4 (1 ):e10205. 10.1002/lrh2.10205 31989029
8. Donabedian A . The quality of care. How can it be assessed? JAMA. 1988;260 (12 ):1743–1748. 10.1001/jama.260.12.1743 3045356
9. Tricco AC , Lillie E , Zarin W , et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med. 2018;169 (7 ):467–473. 10.7326/M18-0850 30178033
10. Fung-Kee-Fung M , Maziak DE , Pantarotto JR , et al. Regional process redesign of lung cancer care: A learning health system pilot project. Curr Oncol. 2018;25 (1 ):59–66. 10.3747/co.25.3719 29507485
11. Ouzzani M , Hammady H , Fedorowicz Z , Elmagarmid A . Rayyan—A web and mobile app for systematic reviews. Syst Rev. 2016;5 (1 ):210. 10.1186/s13643-016-0384-4 27919275
12. ACT Academy . Quality, Service Improvement and Redesign Tools: A model for measuring quality care. Accessed https://www.med.unc.edu/ihqi/wp-content/uploads/sites/463/2021/01/A-Model-for-Measuring-Quality-Care-NHS-Improvement-brief.pdf
13. Woodcock T , Liberati EG , Dixon-Woods M . A mixed-methods study of challenges experienced by clinical teams in measuring improvement. BMJ Qual Saf. 2021;30 (2 ):106–115. 10.1136/bmjqs-2018-009048
14. Abernethy AP , Ahmad A , Zafar SY , Wheeler JL , Reese JB , Lyerly HK . Electronic patient-reported data capture as a foundation of rapid learning cancer care. Med Care. 2010;48 (6 suppl ):S32–S38. 10.1097/MLR.0b013e3181db53a4 20473201
15. Almario CV , Kogan L , van Deen WK , et al. Health economic impact of a multicenter quality-of-care initiative for reducing unplanned healthcare utilization among patients with inflammatory bowel disease. Am J Gastroenterol. 2021;116 (12 ):2459–2464. 10.14309/ajg.0000000000001540 34730561
16. Azar J , Adams N , Boustani M . The Indiana University Center for Healthcare Innovation and Implementation Science: Bridging healthcare research and delivery to build a learning healthcare system. Z Evid Fortbild Qual Gesundhwes. 2015;109 (2 ):138–143. 10.1016/j.zefq.2015.03.006 26028451
17. Bauer MS , Weaver K , Kim B , et al. The Collaborative Chronic Care Model for mental health conditions: From evidence synthesis to policy impact to scale-up and spread. Med Care. 2019;57 (10 suppl 3 ):S221–S227. 10.1097/MLR.0000000000001145 31517791
18. Bhandari RP , Feinstein AB , Huestis SE , et al. Pediatric-Collaborative Health Outcomes Information Registry (Peds-CHOIR): A learning health system to guide pediatric pain research and treatment. Pain. 2016;157 (9 ):2033–2044. 10.1097/j.pain.0000000000000609 27280328
19. Bozkurt S , Paul R , Coquet J , et al. Phenotyping severity of patient-centered outcomes using clinical notes: A prostate cancer use case. Learn Health Syst. 2020;4 (4 ):e10237. 10.1002/lrh2.10237 33083539
20. Johnson LC , Melmed GY , Nelson EC , et al. Fostering collaboration through creation of an IBD learning health system. Am J Gastroenterol. 2017;112 (3 ):406–408. 10.1038/ajg.2017.9 28195179
21. Kamal AH , Miriovsky BJ , Currow DC , Abernethy AP . Improving the management of dyspnea in the community using rapid learning approaches. Chron Respir Dis. 2012;9 (1 ):51–61. 10.1177/1479972311433576 22308555
22. Kilbourne AM , Evans E , Atkins D . Learning health systems: Driving real-world impact in mental health and substance use disorder research. FASEB Bioadv. 2021;3 (8 ):626–638. 10.1096/fba.2020-00124 34377958
23. Marsolo K , Margolis PA , Forrest CB , Colletti RB , Hutton JJ . A digital architecture for A network-based learning health system: Integrating chronic care management, quality improvement, and research. EGEMS (Wash DC). 2015;3 (1 ):1168. 10.13063/2327-9214.1168 26357665
24. Mayo RM , Summey JF , Williams JE , Spence RA , Kim S , Jagsi R . Qualitative study of oncologists’ views on the CancerLinQ rapid learning system. J Oncol Pract. 2017;13 (3 ):e176–e184. 10.1200/JOP.2016.016816 28118106
25. Potter D , Brothers R , Kolacevski A , et al. Development of CancerLinQ, a health information learning platform from multiple electronic health record systems to support improved quality of care. JCO Clin Cancer Inform. 2020;4 :929–937. 10.1200/CCI.20.00064 33104389
26. Rubinstein SM , Warner JL . CancerLinQ: Origins, implementation, and future directions. JCO Clin Cancer Inform. 2018;2 :1–7. 10.1200/CCI.17.00060
27. van Deen WK , Freundlich N , Kwon MH , et al. The reliability of patient self-reported utilization in an inflammatory bowel diseases learning health system. Crohns Colitis 360. 2021;3 (3 ):otab031. 10.1093/crocol/otab031 36776667
28. Weinstock RS , Prahalad P , Rioles N , Ebekozien O . T1D Exchange Quality Improvement Collaborative: A learning health system to improve outcomes for all people with type 1 diabetes. Clin Diabetes. 2021;39 (3 ):251–255. 10.2337/cd21-0032 34421199
29. Zandi PP , Wang Y-H , Patel PD , et al. Development of the National Network of Depression Centers Mood Outcomes Program: A multisite platform for measurement-based care. Psychiatr Serv. 2020;71 (5 ):456–464. 10.1176/appi.ps.201900481 31960777
30. Finlayson SG , Levy M , Reddy S , Rubin DL . Toward rapid learning in cancer treatment selection: An analytical engine for practice-based clinical data. J Biomed Inform. 2016;60 :104–113. 10.1016/j.jbi.2016.01.005 26836975
31. N’Dri LA , Waters DD , Walsh K , Mehta F , Oliver BJ . System-level variation in multiple sclerosis disease-modifying therapy utilization: findings from the Multiple Sclerosis Continuous Quality Improvement research collaborative. Perm J. 2021;25 :25. 10.7812/TPP/21.025
32. McNutt TR , Benedict SH , Low DA , et al. Using big data analytics to advance precision radiation oncology. Int J Radiat Oncol Biol Phys. 2018;101 (2 ):285–291. 10.1016/j.ijrobp.2018.02.028 29726357
33. Feeley TW , Sledge GW , Levit L , Ganz PA . Improving the quality of cancer care in America through health information technology. J Am Med Inform Assoc. 2014;21 (5 ):772–775. 10.1136/amiajnl-2013-002346 24352553
34. Oliver BJ , Walsh K , Messier R , et al. System-level variation in multiple sclerosis care outcomes: Initial findings from the Multiple Sclerosis Continuous Quality Improvement research collaborative. Popul Health Manag. 2022;25 (1 ):46–56. 10.1089/pop.2021.0040 34134513
35. Gravitt PE , Rositch AF , Jurczuk M , et al. Integrative Systems Praxis for Implementation Research (INSPIRE): An implementation methodology to facilitate the global elimination of cervical cancer. Cancer Epidemiol Biomarkers Prev. 2020;29 (9 ):1710–1719. 10.1158/1055-9965.EPI-20-0501 32561563
36. Walsh K , Shah R , Armstrong JK , Moore ES , Oliver BJ . Comparing traditional modeling approaches versus predictive analytics methods for predicting multiple sclerosis relapse. Mult Scler Relat Disord. 2022;57 :103330. 10.1016/j.msard.2021.103330 35158444
37. deRuiter WK , Barker M , Rahimi A , et al. Smoking cessation training and treatment: Options for cancer centres. Curr Oncol. 2022;29 (4 ):2252–2262. 10.3390/curroncol29040183 35448157
38. Tuzzio L , Hanson LR , Reuben DB , et al. Transforming dementia care through pragmatic clinical trials embedded in learning healthcare systems. J Am Geriatr Soc. 2020;68 (suppl 2 ):S43–S48. 10.1111/jgs.16629 32589283
39. Ellis LA , Sarkies M , Churruca K , et al. The science of learning health systems: Scoping review of empirical research. JMIR Med Inform. 2022;10 (2 ):e34907. 10.2196/34907 35195529
40. Excellence J . Learning Health Networks. Accessed https://www.cincinnatichildrens.org/research/divisions/j/anderson-center/learning-networks
41. Wenger EC , Snyder WM . Communities of Practice: The Organizational Frontier [Harvard Business Review]. 2000.
42. Pomare C , Mahmoud Z , Vedovi A , et al. Learning health systems: A review of key topic areas and bibliometric trends. Learn Health Syst. 2022;6 (1 ):e10265. 10.1002/lrh2.10265 35036549
43. Webster P . Big tech companies invest billions in health research. Nat Med. 2023;29 (5 ):1034–1037. 10.1038/s41591-023-02290-y 37055568
44. Batalden M , Batalden P , Margolis P , et al. Coproduction of healthcare service. BMJ Qual Saf. 2016;25 (7 ):509–517. 10.1136/bmjqs-2015-004315
45. Nix M , McNamara P , Genevro J , et al. Learning collaboratives: Insights and a new taxonomy from AHRQ’s two decades of experience. Health Aff. 2018;37 (2 ):205–212. 10.1377/hlthaff.2017.1144
46. Tossaint-Schoenmakers R , Versluis A , Chavannes N , Talboom-Kamp E , Kasteleyn M . The challenge of integrating eHealth into health care: Systematic literature review of the Donabedian model of structure, process, and outcome. J Med Internet Res. 2021;23 (5 ):e27180. 10.2196/27180 33970123
47. Nash DM , Bhimani Z , Rayner J , Zwarenstein M . Learning health systems in primary care: A systematic scoping review. BMC Fam Pract. 2021;22 (1 ):126. 10.1186/s12875-021-01483-z 34162336
