==== Front BMJ OpenBMJ OpenbmjopenbmjopenBMJ Open2044-6055BMJ Publishing Group BMA House, Tavistock Square, London, WC1H 9JR bmjopen-2017-02043710.1136/bmjopen-2017-020437Evidence Based PracticeResearch15061694Quality indicators for responsible use of medicines: a systematic review http://orcid.org/0000-0001-7876-6004Fujita Kenji Moles Rebekah J Chen Timothy F School of Pharmacy, Faculty of Medicine and Health, The University of Sydney, Sydney, New South Wales, Australia Correspondence to Kenji Fujita; kfuj2522@uni.sydney.edu.au2018 16 7 2018 8 7 e02043706 11 2017 26 4 2018 13 6 2018 © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.2018This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.Objective All healthcare systems require valid ways to evaluate service delivery. The objective of this study was to identify existing content validated quality indicators (QIs) for responsible use of medicines (RUM) and classify them using multiple frameworks to identify gaps in current quality measurements. Design Systematic review without meta-analysis. Setting All care settings. Search strategy CINAHL, Embase, Global Health, International Pharmaceutical Abstract, MEDLINE, PubMed and Web of Science databases were searched up to April 2018. An internet search was also conducted. Articles were included if they described medication-related QIs developed using consensus methods. Government agency websites listing QIs for RUM were also included. Analysis Several multidimensional frameworks were selected to assess the scope of QI coverage. These included Donabedian’s framework (structure, process and outcome), the Anatomical Therapeutic Chemical (ATC) classification system and a validated classification for causes of drug-related problems (c-DRPs; drug selection, drug form, dose selection, treatment duration, drug use process, logistics, monitoring, adverse drug reactions and others). Results 2431 content validated QIs were identified from 131 articles and 5 websites. Using Donabedian’s framework, the majority of QIs were process indicators. Based on the ATC code, the largest number of QIs pertained to medicines for nervous system (ATC code: N), followed by anti-infectives for systemic use (J) and cardiovascular system (C). The most common c-DRPs pertained to ‘drug selection’, followed by ‘monitoring’ and ‘drug use process’. Conclusions This study was the first systematic review classifying QIs for RUM using multiple frameworks. The list of the identified QIs can be used as a database for evaluating the achievement of RUM. Although many QIs were identified, this approach allowed for the identification of gaps in quality measurement of RUM. In order to more effectively evaluate the extent to which RUM has been achieved, further development of QIs may be required. quality in health carequality of carequality indicatorsperformance measuresquality assurancequality measurementspecial-featureunlocked ==== Body Strengths and limitations of this study A comprehensive literature search was undertaken across seven databases and government agency websites without restriction of disease categories and care settings. The classification of quality indicators (QIs) was based on multiple frameworks (eg, Donabedian’s framework, the Anatomical Therapeutic Chemical classification system and a validated classification for causes of drug-related problems) for maximum understanding and profiling of the included QIs. Content validated QIs that were developed using consensus methods were only included, and therefore valid QIs might have been excluded during the screening process. Although 5% of this review processes were verified by multiple authors to check for accuracy, most of the classification was undertaken by one author. Introduction Responsible use of medicines (RUM) is an essential element in achieving quality of care for patients and the community. According to the WHO, RUM implies that the activities, capabilities and existing resources of health system stakeholders are aligned to ensure patients receive the right medicines at the right time, use them appropriately and benefit from them.1 RUM, however, is not easily achievable, and if medicines are used inappropriately, negative consequences for both patients and/or the society may occur. It is reported that worldwide more than 50% of all medicines are prescribed, dispensed or sold inappropriately, while 50% of patients fail to take them correctly.2 In addition, it has been reported that one-third of preventable drug-related admissions are associated with medication non-adherence, 31% are related to prescribing problems and 22% are related to monitoring problems.3 The frequency of these medication errors varies depending on the specific medicine. For example, previous systematic reviews have found that preventable drug-related admissions to hospital accounted for 3.7% of all admissions, of which four groups of drugs, antiplatelets, diuretics, non-steroidal anti-inflammatory and anticoagulants accounted for more than 50% of the drug groups associated with those preventable drug-related hospitalisations.3 From the economic perspective, globally, the cost associated with medication errors has been estimated at US$42 billion annually or almost 1% of total global health expenditure.4 Given the health concerns and the economic burden associated with medication errors, the achievement of RUM underpinned by an evidence-based approach has become increasingly important worldwide. One critical element for any healthcare system or organisation is how to measure and evaluate RUM. A widely used method to do this is the use of quality indicators (QIs).5 6 QIs are explicitly defined and measurable items referring to the structures, processes or outcomes of care are usually described with a denominator and a numerator.7 The denominator is the total number of cases in the intended population, and the numerator is the number of cases that fulfil a predetermined criterion, and the calculated QI score indicates the quality of care.8 QIs can be used to monitor the quality of care provided by healthcare professionals in a single institution, to promote quality improvement activities, to make comparisons over time between institutions or to support consumers to choose healthcare providers.5 For QIs to be useful, they must be developed with scientific rigour, and all quality dimensions of care must be measured to capture a comprehensive landscape of healthcare quality.5 To achieve RUM using QIs, it is first necessary to identify existing QIs for RUM, independent of disease categories and care settings. Additionally, in the light of the concept of RUM, multifaceted assessment is required to gain full understanding of the breadth of coverage by QIs. To our knowledge, however, previously conducted systematic reviews have been restricted to setting (eg, hospital),9 disease state (eg, HIV/AIDS),10 specific to a healthcare group (eg, nursing sensitive QIs) or indicator name (eg, clinical indicators)11 and have only been classified based on Donabedian’s framework or implicit frameworks such as quality dimensions defined by the Institute of Medicine.12 Hence, the main purpose of this systematic review was to identify existing content validated QIs for RUM independent of disease category and care settings, and then classify them using multiple frameworks in order to identify gaps in current quality measurements. Methods Data sources This systematic review was performed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (see online supplementary table S1).13 Two approaches were used to identify relevant QIs. 10.1136/bmjopen-2017-020437.supp1Supplementary file 1 First, CINAHL, Embase, Global Health, International Pharmaceutical Abstract, MEDLINE, PubMed and Web of Science databases were searched to identify relevant articles published up to 5 April 2018. No restriction on year of study was applied. Search strategies comprised keywords and, when available, controlled vocabulary such as Medical Subject Headings/EMTREE based on three main terms: ‘quality indicators’, ‘development’ and ‘consensus’. Since ‘quality indicators’ are referred to by wide variety of terms such as clinical indicators, or performance measures, the finalised search strategies were developed using an iterative development process during which citations identified by various search terms were screened for relevance. We chose ‘consensus’ as a main term because QIs are recommended to be developed using expert panels based on rigorous evidence in order to ensure high face validity and content validity.14 Exact search dates for each database with the search strategies are included in online supplementary table S2. 10.1136/bmjopen-2017-020437.supp2Supplementary file 2 Second, using Google, an internet search was also conducted (search terms: quality indicators, clinical indicators, performance indicator or performance measures) to capture additional QIs listed in the websites of relevant organisations responsible for quality improvement. Potentially relevant organisation’s websites, found in the process of literature review,9 12 15–17 were also searched (see online supplementary table S3). 10.1136/bmjopen-2017-020437.supp3Supplementary file 3 Study selection Inclusion criteria Articles were included if they fulfilled the following criteria: (A) the article was peer reviewed and published in English, (B) numerators and denominators were defined for the QIs, or they could be directly deduced from the descriptions of the QIs, (C) the publication contained at least one medication-related QI, (D) the development of QIs was one of the objectives and (E) QIs were developed using consensus methods in order to confirm content validity. Furthermore, relevant organisations’ QIs found from websites were included if the organisation was a government agency for ensuring quality in healthcare, and at least one QI for RUM was reported with a clear description, as detailed above (B). Given the concept of QIs and RUM mentioned above, we regarded a measurement tool as a QI for RUM when the definition of the QI referred to a medication. In addition, if publications concerned the same project/QIs set, the descriptions of the QIs in the most recent publication were used for data extraction. Exclusion criteria Articles were excluded if the consensus results for QI development were unclear, if QI lists were obtainable only by purchase or if QIs were for monitoring the effectiveness of national policies. This study selection process was performed using a purposed designed screening proforma (see online supplementary table S4). The retrieved articles were transferred into Endnote to remove duplicates, then initial screening of journal names, titles and abstracts was conducted to remove irrelevant articles. 10.1136/bmjopen-2017-020437.supp4Supplementary file 4 Data extraction One researcher (KF) extracted the following data from the full text of included articles or websites: publication year, country or other targeted location in which QIs were intended to be used, name of measurement tools, total number of QIs, the number of relevant QIs for RUM, scope of the QIs and definition of QIs (numerator and denominator, if available). A data extraction proforma was designed, pilot-tested on five included studies, then refined accordingly. Analysis Descriptive statistics were computed for the results of the present review based on counts and proportions where relevant. Since the components of RUM are multidimensional, multiple frameworks were used to understand the breadth of coverage by QIs. That is, we used four types of classification: (1) problem type; (2) Donabedian’s framework; (3) the Anatomical Therapeutic Chemical (ATC) classification system; and (4) causes of drug-related problems (c-DRPs) classification system. Problem type The first step of a structured QI development process is to identify the problem for which measurement is needed.18 Classifying QIs according to problem type can highlight prioritised problems for QI development. Therefore, QI sets described in each source were classified into the following six problem types proposed by Evans et al18:Disease based: problems relevant to diseases, illnesses, conditions, injuries or procedures for which the quality of care needs to be measured. Patient based: problems related to patient groups, such as vulnerable elders and paediatric patients. Treatment modality based: problems relevant to service providing areas, such as intensive care units or palliative care settings. Organisation based: problems relevant to organisational issues, such as whether organisations have effective structures in place at an organisational level to support quality and safety. Generic problems: problems relevant to issues that are multidisciplinary in nature and relevant to any form of healthcare delivery in multiple physical settings, such as falls prevention, or pain management. Profession based: problems unique to the different healthcare professions and include availability and competence of healthcare personnel. If a QI set related to more than one problem type, they were classified accordingly (eg, an article about QIs for nursing practice in the operating room fell into treatment modality-based and profession-based problem). Donabedian’s framework QIs were classified according to the widely used Donabedian’s framework of structure (referred to the factors that designate the conditions under which care is provided, such as material or human resources), process (referred to the actions of healthcare professionals, such as prescribing or monitoring) or outcome (referred to the changes in individuals that can be attributed to care provided), irrespective of the category defined in the original source.19 Online supplementary table S5 lists examples of QIs classified into these three categories. 10.1136/bmjopen-2017-020437.supp5Supplementary file 5 The ATC classification system QIs were first classified into medicine class specific indicators or general medication indicators, depending on whether the definition of the QI described a specific class of medicines. For example, a QI ‘numerator: patients with acute myocardial infarction (AMI) received aspirin within 3 hours of hospital arrival/denominator: AMI patients without aspirin contraindications’20 was classified as a medicine class specific indicator, while a QI ‘numerator: number of patients aged 65 years and older whose current medications are documented and reconciled at admission/denominator: number of patients aged 65 years and older in sample’21 was classified as a general medication indicator. After this process, medicine class specific indicators were classified using the first and second levels of the ATC code.22 A single QI was sometimes allocated into more than one ATC code. For example, a QI, ‘percentage of patients using opioids with concomitant laxatives’,23 represented A06 (drugs for constipation) and N02 (analgesics). c-DRPs classification system Since minimising the factors that contribute to drug-related problems (ie, causes of DRPs) is closely linked to achieving RUM, the extracted QIs were classified using a comprehensive taxonomy of the causes of DRPs.24 This taxonomy divides c-DRPs into the following nine categories.Drug selection, for example, whether appropriate drugs are selected by healthcare professionals. Drug form, for example, whether appropriate drug forms are selected by healthcare professionals. Dose selection, for example, whether appropriate drug dosages are selected by healthcare professionals. Treatment duration, for example, whether drugs are being prescribed or dispensed for an appropriate duration by healthcare professionals. Drug use process, for example, whether drugs are taken properly by patients. Logistics, for example, whether necessary drugs are properly delivered to the patients. Monitoring, for example, monitoring for the effect/adverse effects of drugs. Adverse drug reactions, for example, the occurrence of adverse drug reactions. Other. Note that a single QI was sometimes allocated into more than one c-DRP category. Online supplementary table S6 illustrates how QIs were classified using the c-DRP taxonomy. 10.1136/bmjopen-2017-020437.supp6Supplementary file 6 All processes were conducted independently by one author (KF), and 5% of these processes were verified by TFC and RJM. Any issues that arose during the process were resolved by discussion between the research team (KF, RJM and TFC). Meta-analysis was not applicable due to heterogeneity in interventions, methods and reported outcomes. We believed that it was not necessary to assess the quality of the content validated QIs included in our studies such as their feasibility, and reliability because problems affecting QIs (eg, feasibility of data collection, reliability of calculating QI scores and opportunities for gaming) vary depending on the healthcare infrastructure and healthcare remuneration system in each country. Patient and public involvement As this was a literature review, there was no patient and public involvement in this study. Results Study selection Initially, a total of 39 430 articles were obtained. The sample included 17 822 duplicate records, which were removed. After the initial screening, 973 full texts were assessed for eligibility with 842 excluded based on the inclusion and exclusion criteria. Eventually 131 articles met all inclusion criteria and were included in our review. Additionally, through the internet search, five relevant websites were identified and included in our review (figure 1). Figure 1 Study flow diagram. QI, quality indicator; RUM, responsible use of medicines. Study characteristics Of the 131 articles, 78 articles (60%) developed QIs for use in three countries: USA (n=36),25–60 Canada (n=26)61–86 and Netherlands (n=16).23 87–101 The remaining 53 articles developed QIs for use in 16 other countries20 21 102–145 and 4 other targeted locations (such as the Organisation for Economic Co-operation and Development (OECD) countries)146–152 (figure 2). Of the five relevant websites, three were Australian organisations,153–155 one was a UK organisation156 and the other was USA organisation.157 The three Australian and UK organisations developed QIs at the organisation level, while the American website, National Quality Measures Clearinghouse, sponsored by the Agency for Healthcare Research and Quality, stored QIs developed by various countries. Of 7750 QIs listed in the 131 articles and 5 websites, we identified 2431 QIs for RUM: 1947 QIs from journal articles and 484 QIs from the web. Figure 2 The number of publications by country and other target location. While there were 21 different ways of labelling the measurement tools, ‘quality Indicators’ (n=80, 59%) was the most commonly used term in our included articles and websites, followed by ‘quality measures’ (n=11, 8%), ‘quality of care indicators’ (n=8, 6%) and ‘indicators’ (n=7, 5%). In terms of the problem type, 43% of QI sets pertained to disease-based problems (n=89, eg, knee osteoarthritis), then 27% for treatment modality-based problems (n=55, eg, primary care), 21% for patient-based problems (n=44, eg, geriatric care), 5% for profession based problems (n=11, eg, community pharmacists), 3% generic problems (n=6, eg, long-term prescribing) or 1% organisation-based practice (n=2, eg, centralised intake systems). The majority of QIs (n=2289, 94%) were process indicators, while structure (n=80) and outcome (n=62) indicators accounted for only 3% each (table 1). Table 1 Characteristics of studies and quality indicator sets Reference Year Country/other target location QI name Scope of QIs (setting, condition, target patient group, occupation) Problem type* QIs for RUM (%) Donabedian’s framework† QI type‡ S P O Me Ge Broccoli et al102 2018 Africa Quality indicators Emergency care T 19/76 (25) 5 14 0 16 3 Tropea et al21 2011 Australia Clinical indicators Older hospitalised patients P 4/19 (21) 0 4 0 1 3 ACSQHC153 § 2012 Australia Practice-level indicators Primary care T 3/35 (9) 0 3 0 0 3 ACSQHC154 § 2014 Australia National Quality Use of Medicines Indicators Hospital care T 35/37 (95) 2 33 0 25 10 Caughey et al103 2014 Australia Medication-related indicators Primary care T 28/28 (100) 0 28 0 28 0 Victorian Government155 § 2015 Australia Quality indicators Residential aged care P, T 1/5 (20) 0 1 0 0 1 Nag et al104 2016 Australia Quality indicators Prostate cancer D 1/12 (8) 0 1 0 1 0 O’Connor et al105 2017 Australia Quality indicators Psychotropic prescribing for people with dementia in aged psychiatry inpatient units D, P, T 6/6 (100) 0 6 0 6 0 Sibthorpe et al106 2017 Australia Indicators Otitis media in primary healthcare for Aboriginal and Torres Strait Islander children D, P, T 2/12 (17) 0 2 0 2 0 Stordeur et al107 2012 Belgium Quality indicators Breast cancer D 9/32 (28) 0 9 0 9 0 Grypdonck et al108 2014 Belgium Quality indicators Knee osteoarthritis D 8/21 (38) 0 8 0 8 0 Stordeur et al109 2015 Belgium Quality indicators Oesophageal cancer and gastric cancer D 4/29 (14) 0 4 0 4 0 De Schreye et al110 2017 Belgium Quality indicators End-of-life care in people with Alzheimer’s disease, cancer or chronic obstructive pulmonary disease D, P 23/81 (28) 0 23 0 23 0 Leemans et al111 2017 Belgium Quality indicators Palliative care P 4/31 (13) 0 2 2 4 0 de Carvalho et al112 2017 Brazil Indicators Adult intensive care P, T 7/62 (11) 7 0 0 7 0 Mackinnon and Hepler61 2002 Canada Clinical indicators Geriatric care P 52/52 (100) 0 52 0 52 0 Robertson and MacKinnon62 2002 Canada Clinical indicators Geriatric care P 52/52 (100) 0 52 0 52 0 Burge et al63 2007 Canada Quality indicators Cardiovascular primary care D, T 11/31 (35) 0 11 0 11 0 Kröger et al64 2007 Canada Quality indicators Older adults with cognitive impairment or dementia D, P 21/72 (29) 0 21 0 9 12 Ko et al65 2008 Canada Quality indicators Percutaneous coronary intervention D 7/26 (27) 2 5 0 7 0 MacKinnon et al66 2008 Canada Clinical indicators Type 2 diabetes D 12/21 (57) 0 12 0 12 0 Nigam et al67 2008 Canada Medication use safety indicators Medication use at inpatient and outpatient settings T 20/20 (100) 1 17 2 6 14 Tu et al68 2008 Canada Quality indicators Acute myocardial infarction D 16/38 (42) 0 16 0 16 0 Dixon et al69 2009 Canada Quality indicators Patients undergoing hepatic resection for metastatic colorectal cancer D 2/18 (11) 0 2 0 2 0 Teresato and Lougheed70 2010 Canada Performance indicators Primary care for asthma D, T 5/20 (25) 0 5 0 5 0 Krzyzanowska et al71 2011 Canada Quality indicators Cancer care for women D, P 5/31 (16) 0 5 0 5 0 Schull et al72 2011 Canada Quality of care indicators Emergency department care T 16/48 (33) 0 16 0 16 0 Addington et al73 2012 Canada Performance measures Schizophrenia D 6/36 (17) 0 6 0 6 0 Stang et al74 2013 Canada Quality indicators High acuity paediatric conditions D, P 28/62 (45) 0 28 0 28 0 Darling et al75 2014 Canada Quality indicators Non-small cell lung cancer operations D 1/17 (6) 0 1 0 1 0 Nguyen et al76 2014 Canada Quality indicators Inflammatory bowel disease D 6/11 (55) 0 6 0 6 0 Santana and Stelfox77 2014 Canada Quality indicators Adult injury care D, P 4/31 (13) 1 3 0 4 0 Barber et al78 2015 Canada Key performance indicators Centralised intake systems for patients with osteoarthritis and rheumatoid arthritis D, O 2/28 (7) 0 2 0 2 0 Fernandes et al79 2015 Canada Clinical pharmacy key performance indicators Hospital pharmacists Pr 8/8 (100) 0 8 0 0 8 Khare et al80 2016 Canada Performance indicators Breast, prostate, colorectal and lung cancer D 17/78 (22) 0 12 5 17 0 McKelvie et al81 2016 Canada Quality indicators Heart failure D 1/6 (17) 0 1 0 1 0 Khare et al82 2017 Canada Quality indicators Bladder cancer D 22/60 (37) 0 19 3 22 0 Tu et al83 2017 Canada Performance indicators Primary prevention of cardiovascular disease in the ambulatory care D, T 9/28 (32) 0 9 0 9 0 Tu et al84 2017 Canada Quality indicators Chronic kidney disease in primary care D, T 6/17 (35) 0 6 0 6 0 Chartrand et al85 2018 Canada Quality indicators Oral anticoagulant management in community pharmacies D, Pr 38/38 (100) 2 34 2 38 0 Mukerji et al86 2018 Canada Quality indicators Ambulatory diabetes care D, T 18/35 (51) 0 18 0 16 2 Sun et al20 2011 China Quality indicators Acute myocardial infarction D 10/23 (43) 0 10 0 10 0 Bao et al113 2015 China Quality indicators Breast cancer D 11/31 (35) 0 11 0 11 0 Chen et al114 2016 China Quality indicators Neonatal intensive care units nursing Pr, T 2/11 (18) 0 2 0 1 1 Wu et al115 2016 China Quality indicators Nursing practice in the operating room Pr, T 3/23 (13) 1 1 1 2 1 Li et al116 2017 China Indicators Rational drug use for community-acquired pneumonia in children D, P 44/44 (100) 0 44 0 42 2 Wang et al117 2017 China Quality indicators Non-small cell lung cancer care D 10/21 (48) 0 10 0 10 0 Ju et al118 2018 China Quality indicators Emergency nursing care Pr, T 5/16 (31) 0 5 0 3 2 Tang et al119 2018 China Quality indicators Home care T 1/70 (1) 1 0 0 0 1 Saust et al120 2017 Denmark Quality indicators Diagnosis and antibiotic treatment of acute respiratory tract infections in general practice D, T 19/31 (61) 0 19 0 19 0 Campbell et al146 2008 Europe Quality indicators Cardiovascular disease in primary care D, T 10/44 (23) 0 10 0 8 2 Adriaenssens et al147 2011 Europe Quality indicators Outpatient antibiotic prescribing G, P 21/21 (100) 0 21 0 21 0 Petersson et al148 2014 Europe Healthcare quality indicators Rheumatoid arthritis D 1/14 (7) 0 1 0 1 0 Boulkedid et al121 2013 France Quality indicators Obstetrical care in maternity units P, T 2/18 (11) 0 2 0 2 0 Follmann et al122 2014 Germany Quality indicators Melanoma D 3/12 (25) 0 3 0 3 0 Hussein et al123 2017 Germany Quality indicators Systemic antibiotics in dentistry D, Pr 12/12 (100) 0 12 0 12 0 Hermann et al149 2006 International Quality indicators Mental health D 4/12 (33) 0 4 0 4 0 Barber et al150 2015 International Quality indicators Cardiovascular disease care in patients with rheumatoid arthritis D 2/11 (18) 0 2 0 2 0 Wakai et al124 2013 Ireland Key performance indicators Performance of emergency department T 15/97 (15) 1 14 0 14 1 Murphy et al125 2016 Ireland Key performance indicators Prehospital emergency care T 19/101 (19) 0 18 1 19 0 Barry et al126 2016 Ireland and UK Prescribing indicators Prescribing for children in primary care P, T 12/12 (100) 0 12 0 12 0 Fukuma et al127 2016 Japan Quality indicators Non-dialysis chronic kidney disease D 4/11 (36) 0 4 0 4 0 Masaki et al128 2017 Japan Quality indicators Elderly end-of-life care in nursing P, Pr 4/33 (12) 0 4 0 4 0 Ueda et al129 2017 Japan Quality indicators Low-risk labour care provided by midwives P, Pr 2/23 (9) 0 2 0 1 1 Ntoburi et al151 2010 Low-income countries Indicators Paediatric inpatient care P, T 56/112 (50) 26 30 0 55 1 Perez-Cuevas et al130 2012 Mexico Quality of care indicators Type 2 diabetes D 4/18 (22) 0 4 0 4 0 Doubova et al131 2014 Mexico Quality indicators Antenatal care P 2/14 (14) 0 2 0 2 0 Muijrers et al87 2004 Netherlands Prescribing indicators General practice T 34/34 (100) 0 34 0 34 0 Mourad et al88 2007 Netherlands Quality indicators Subfertility care D 8/39 (21) 0 8 0 8 0 Drašković et al89 2008 Netherlands Quality indicators Clinical practice at memory clinics T 1/14 (7) 0 1 0 1 0 Martirosyan et al90 2008 Netherlands Prescribing quality indicators Type 2 diabetes D 14/14 (100) 0 14 0 14 0 van der Ploeg et al91 2008 Netherlands Quality indicators General practice care for vulnerable elders P, T 36/81 (44) 0 36 0 24 12 Perry et al92 2010 Netherlands Quality indicators Dementia D 2/23 (9) 0 2 0 2 0 Stienen et al93 2011 Netherlands Quality indicators Paediatric constipation D, P 3/7 (43) 0 3 0 3 0 Wierenga et al94 2011 Netherlands Quality indicators Inhospital pharmaceutical care for elderly patients P, Pr, T 85/87 (98) 0 85 0 71 14 Luitjes et al95 2013 Netherlands Quality indicators Hypertensive diseases in pregnancy D, P 5/14 (36) 0 5 0 5 0 van den Bosch et al96 2014 Netherlands Quality indicators Antimicrobial use in hospitalised adult patients with sepsis D, P, T 5/5 (100) 0 5 0 5 0 van den Bosch et al97 2015 Netherlands Quality indicators Antibiotic use in hospitalised adults D, P, T 11/11 (100) 2 9 0 11 0 Woiski et al98 2015 Netherlands Quality indicators Postpartum haemorrhage D 7/22 (32) 1 6 0 7 0 Hommel et al99 2016 Netherlands Quality indicators Perioperative diabetes care D 7/36 (19) 1 6 0 7 0 Smits et al100 2016 Netherlands Prescribing quality indicators Chronic kidney disease D 16/16 (100) 0 16 0 16 0 Teichert et al23 2016 Netherlands Quality indicators Pharmaceutical care in community pharmacies Pr, T 67/67 (100) 21 43 3 23 44 Smits et al101 2017 Netherlands Prescribing quality indicators Type 2 diabetes in primary care D, T 20/20 (100) 0 20 0 20 0 Idänpään-Heikkilä et al152 2006 OECD countries Quality indicators Cardiac care D 8/17 (47) 0 8 0 8 0 Petek et al132 2012 Slovenia Quality indicators Cardiovascular disease prevention for primary care D, T 14/88 (16) 0 14 0 12 2 Minaya-Muñoz et al133 2013 Spain Quality measures Lateral epicondylalgia D 2/12 (17) 0 2 0 2 0 Calvet et al134 2014 Spain Quality indicators Inflammatory bowel disease D 14/56 (25) 0 14 0 14 0 Ruiz-Canela-Cáceres et al135 2015 Spain Quality indicators Childhood asthma in primary care D, P, T 2/7 (29) 0 2 0 2 0 Soria-Aledo et al136 2016 Spain Indicators General surgery T 2/13 (15) 0 2 0 2 0 Bianchi et al137 2013 Switzerland Quality indicators Colorectal cancer D 7/27 (26) 0 6 1 7 0 Chung et al138 2008 Taiwan Performance measures Breast cancer D 2/15 (13) 0 2 0 2 0 Chung et al139 2010 Taiwan Core measures Colorectal cancer D 3/17 (18) 0 3 0 3 0 Cantrill et al140 1998 UK Indicators Long-term prescribing in general practice G, T 9/9 (100) 0 9 0 1 8 Morris and Cantrill141 2003 UK Quality indicators Preventing drug-related morbidity in primary care G, T 24/24 (100) 0 24 0 24 0 Steel et al142 2004 UK Quality indicators Healthcare of older adults in primary and secondary care P 40/102 (39) 0 40 0 37 3 Tully et al143 2005 UK Indicators Long term prescribing in primary and secondary care G, T 14/14 (100) 0 14 0 0 14 Gill et al144 2014 UK Quality indicators Child healthcare in general practice P 10/35 (29) 0 10 0 9 1 Spencer et al145 2014 UK Prescribing safety indicators Safety of prescribing in general practice G, T 56/56 (100) 0 56 0 56 0 NICE156 3 § 2016 UK NICE indicators General practice T 33/125 (26) 0 30 3 33 0 Hadorn et al25 1996 USA Review criteria Heart failure D 4/8 (50) 0 4 0 3 1 Asch et al26 2001 USA Quality of care indicators Women with hypertension D, P 6/13 (46) 0 6 0 5 1 Mikuls et al27 2004 USA Quality of care indicators Gout D 9/10 (90) 0 9 0 9 0 Saliba et al28 2004 USA Quality indicators Nursing home residents T 54/114 (47) 1 53 0 54 0 Krumholz et al29 2006 USA Performance measures ST-elevation and non-ST-elevation myocardial infarction D 8/11 (73) 0 8 0 8 0 McGory et al30 2006 USA Quality indicators Patients undergoing colorectal cancer surgery D 20/92 (22) 0 20 0 19 1 Mularski et al31 2006 USA Quality measures Palliative care T 1/18 (6) 0 1 0 1 0 Mangione-Smith et al32 2007 USA Quality indicators Paediatric outpatients P, T 58/175 (33) 0 58 0 58 0 Smith et al33 2007 USA Quality indicators Home-based primary care T 71/200 (36) 1 70 0 59 12 Wenger et al34 2007 USA Quality indicators Vulnerable elders P 146/392 (37) 0 146 0 129 17 Estes et al35 2008 USA Performance measures Outpatient adults with non-valvular atrial fibrillation or atrial flutter D, P, T 2/3 (67) 0 2 0 2 0 Bilimoria et al36 2009 USA Quality indicators Pancreatic cancer D 3/43 (7) 1 2 0 3 0 Lorenz et al37 2009 USA Quality measures Supportive cancer care D 36/92 (39) 0 36 0 35 1 McGory et al38 2009 USA Quality indicators Perioperative care for elderly surgical patients P, T 17/91 (19) 0 17 0 13 4 Yazdany et al39 2009 USA Quality indicators Systemic lupus erythematosus D 14/20 (70) 0 14 0 14 0 Cheng et al40 2010 USA Quality indicators Multiple sclerosis D 19/76 (25) 0 19 0 17 2 Kanwal et al41 2010 USA Quality indicators Nonvariceal upper gastrointestinal haemorrhage D 6/26 (23) 0 6 0 6 0 Kanwal et al42 2010 USA Quality indicators Cirrhosis D 12/41 (29) 0 12 0 12 0 Schenck et al43 2010 USA Quality measures Hospice and palliative care T 7/34 (21) 0 7 0 7 0 Khanna et al44 2011 USA Quality indicators Systemic sclerosis D 10/32 (31) 0 10 0 10 0 SooHoo et al45 2011 USA Quality of care indicators Patients undergoing total hip or total knee replacement D 8/68 (12) 0 8 0 5 3 Wang et al46 2011 USA Quality of care indicators Children with sickle cell disease D, P 13/41 (32) 0 13 0 13 0 Anger et al47 2013 USA Quality of care indicators Women with urinary incontinence D, P 7/27 (26) 0 7 0 7 0 Jackson et al48 2013 USA Quality indicators Colorectal cancer D 11/34 (32) 0 11 0 11 0 Melmed et al49 2013 USA Quality indicators Inflammatory bowel disease D 8/21 (38) 0 5 3 8 0 Wang et al50 2013 USA Quality of care indicators Infantile spasms D, P 10/21 (48) 0 10 0 10 0 Yadlapati et al51 2015 USA Quality measures Gastro-oesophageal reflux disease D 15/25 (60) 0 15 0 15 0 Faro et al52 2016 USA Quality indicators Follow-up care for individuals with positive screens for sickle cell disease and trait D 2/9 (22) 0 2 0 2 0 Vila et al53 2016 USA Quality measures Adult cochlear implant centres D, P, T 2/8 (25) 0 1 1 2 0 Yazdany et al54 2016 USA Quality measures Rheumatoid arthritis D 1/4 (25) 0 1 0 1 0 Chowdhury et al55 2017 USA Quality metrics Adult congenital heart disease and paediatric cardiology care D, P 5/27 (19) 0 4 1 5 0 Hepner et al56 2017 USA Quality measures Unhealthy alcohol use D 3/25 (12) 0 3 0 3 0 Ingraham et al57 2017 USA Quality indicators Emergency general surgery care T 3/25 (12) 2 1 0 2 1 Mangione-Smith et al58 2017 USA Quality indicators Hospital-based care for common paediatric respiratory illnesses D, P, T 43/76 (57) 0 43 0 43 0 Odetola et al59 2017 USA Quality measures Inhospital care of paediatric sepsis syndrome D, P, T 3/7 (43) 1 2 0 3 0 NQMC157 § 2018 USA Quality measures Various care All 412/2525 (16) 0 378 34 376 36 Parast et al60 2018 USA Quality measures Hospital-based care for suicidal youth D, P, T 4/4 (100) 0 4 0 2 2 Total 2431/7750 (31) 60 2289 62 2184 247 *Problem type: D, disease based; G, generic; O, organisation based; P, patient based; Pr, profession based; T, treatment modality based. †Donabedian’s framework: S, structure; P, process; O, outcome. ‡QI type: Me, medicine class specific; Ge, general medication. §Government agency website. ACSQHC, The Australian Commission on Safety and Quality in Healthcare; NICE, National Institute for Health and Clinical Excellence; NQMC, National Quality Measures Clearinghouse; QI, quality indicator; RUM, responsible use of medicines. Of 2431 QIs, 247 QIs (10%) were general medication indicators, and 2184 QIs (90%) were medicine class specific indicators. Some of the 2184 QIs represented more than one ATC code resulting in 2613 first level of ATC classifications. Of these, the most number of QIs covered medicines for nervous system (N, n=407, 16%), followed by the anti-infectives for systemic use (J, n=397, 15%), cardiovascular system (C, n=364, 14%) and blood and blood forming organs (B, n=345, 13%) (figure 3). Dermatological medicines (D) were covered by the least number of QIs (n=19, 0.7%) aside from antiparastic products, insecticides and repellents (P, n=7, 0.3%). Figure 3 The number of QIs by first-level ATC code. ATC, Anatomical Therapeutic Chemical; QIs, quality indicators. The distribution of the QIs across the second level of ATC code and c-DRPs classification system is presented in table 2. General medication indicators were only classified using c-DRPs category. Because some QIs represented more than one ATC code and/or c-DRPs category, the total number of the QIs contained within each cell of the matrix was 3666. Of these, when investigating the number of QIs in each c-DRPs category, the largest number of QIs for ‘drug selection’ pertained to antibacterials for systemic use (J01, 176 of 2117, 8%), followed by antithrombotic agents (B01, 172 of 2117, 8%). Antithrombotic agents (B01) also contributed the largest number of QIs for ‘dose selection’ (20 of 142, 14%) and the ‘drug use process’ (52 of 439, 12%) and ‘monitoring’ (52 of 574, 9%). Likewise, the most number of QIs for ‘treatment duration’ (13 of 85, 15%) pertained to psychoanaleptics (N06). Table 2 Distribution of QIs for RUM by the ATC code (rows) and the c-DRPs category (columns)* ATC code 1. Drug selection 2. Drug form 3. Dose selection 4. Treatment duration 5. Drug use process 6. Logistics 7. Monitoring 8. Adverse drug reaction 9. Other Total, n (%) A: alimentary tract and metabolism  A01: Stomatological preparations 7 5 12 (0.3)  A02: Drugs for acid related disorders 37 3 2 8 1 51 (1.4)  A03: Drugs for functional gastrointestinal disorders 35 3 4 42 (1.1)  A04: Antiemetics and antinauseants 23 4 5 32 (0.9)  A06: Drugs for constipation 20 1 3 2 26 (0.7)  A07: Antidiarrheals, intestinal antiinflammatory/antiinfective agents 14 1 2 1 1 2 21 (0.6)  A08: Antiobesity preparations, excl. diet products 1 1 (0)  A10: Drugs used in diabetes 40 1 4 19 3 19 2 88 (2.4)  A11: Vitamins 28 1 1 1 1 32 (0.9)  A12: Mineral supplements 24 1 1 5 3 34 (0.9) B: Blood and blood forming organs  B01: Antithrombotic agents 172 20 8 52 6 52 1 9 320 (8.7)  B02: Antihemorrhagics 2 1 3 (0.1)  B03: Antianemic preparations 9 1 1 3 2 1 17 (0.5)  B05: Blood substitutes and perfusion solutions 33 1 9 2 4 1 5 55 (1.5) C: Cardiovascular system  C01: Cardiac therapy 30 1 4 4 4 15 58 (1.6)  C02: Antihypertensives 50 5 1 2 10 68 (1.9)  C03: Diuretics 57 6 2 3 23 1 92 (2.5)  C04: Peripheral vasodilators 5 1 6 (0.2)  C05: Vasoprotectives 1 1 (0)  C07: Beta blocking agents 108 5 5 11 10 139 (3.8)  C08: Calcium channel blockers 48 6 5 10 69 (1.9)  C09: Agents acting on the renin-angiotensin system 128 7 2 7 1 34 179 (4.9)  C10: Lipid modifying agents 53 2 1 7 8 71 (1.9) D: Dermatologicals  D01: Antifungals for dermatological use 2 2 (0.1)  D02: Emollients and protectives 3 3 (0.1)  D06: Antibiotics and chemotherapeutics for dermatological use 4 4 (0.1)  D07: Corticosteroids, dermatological preparations 4 1 5 (0.1)  D08: Antiseptics and disinfectants 1 1 (0)  D10: Anti-acne preparations 1 3 4 (0.1)  D11: Other dermatological preparations 1 1 (0) G: Genito urinary system and sex hormones  G01: Gynecological antiinfectives and antiseptics 4 1 2 1 8 (0.2)   G02: Other gynecologicals 2 2 (0.1)  G03: Sex hormones and modulators of the genital system 29 9 8 46 (1.3)  G04: Urologicals 16 5 2 23 (0.6) H: Systemic hormonal preparations, excl. sex hormones and insulins  H01: Pituitary and hypothalamic hormones and analogues 8 1 1 2 5 17 (0.5)  H02: Corticosteroids for systemic use 53 5 3 5 9 3 78 (2.1)  H03: Thyroid therapy 2 2 2 5 11 (0.3)  H05: Calcium homeostasis 14 3 17 (0.5) J: Antiinfectives for systemic use  J01: Antibacterials for systemic use 176 7 9 12 38 8 22 8 280 (7.6)  J02: Antimycotics for systemic use 2 1 3 (0.1)  J04: Antimycobacterials 2 1 3 (0.1)  J05: Antivirals for systemic use 12 2 2 6 4 3 29 (0.8)  J07: Vaccines 94 10 1 5 6 116 (3.2) L: Antineoplastic and immunomodulating agents  L01: Antineoplastic agents 94 3 5 35 3 45 3 11 199 (5.4)  L02: Endocrine therapy 15 1 4 4 24 (0.7)  L03: Immunostimulants 4 2 6 (0.2)  L04: Immunosuppressants 23 4 1 6 2 12 48 (1.3) M: Musculoskeletal system  M01: Antiinflammatory and antirheumatic products 53 5 6 14 1 18 2 1 100 (2.7)  M02: Topical products for joint and muscular pain, 45 5 5 13 1 18 2 1 90 (2.5)  M03: Muscle relaxants 20 1 1 3 25 (0.7)  M04: Antigout preparations 10 3 3 16 (0.4)  M05: Drugs for treatment of bone diseases 27 1 1 3 32 (0.9) N: Nervous system  N01: Anesthetics 12 3 2 1 1 19 (0.5)  N02: Analgesics 82 1 9 1 24 5 24 3 3 152 (4.1)  N03: Antiepileptics 19 2 8 5 5 2 41 (1.1)  N04: anti-Parkinson drugs 30 1 4 5 40 (1.1)  N05: Psycholeptics 72 6 8 12 5 28 2 3 136 (3.7)  N06: Psychoanaleptics 73 8 13 9 2 29 2 136 (3.7)  N07: Other nervous system drugs 11 12 2 25 (0.7)  P: Antiparasitic products, insecticides and repellents (p01: antiprotozoals) 2 1 2 1 3 R: Respiratory system  R01: Nasal preparations 5 2 7 (0.2)  R03: Drugs for obstructive airway diseases 69 3 3 37 4 13 2 131 (3.6)  R05: Cough and cold preparations 6 1 7 (0.2)  R06: Antihistamines for systemic use 31 2 3 36 (1) S: Sensory organs  S01: Ophthalmologicals 23 5 3 2 33 (0.9)  S02: Otologicals 2 2 (0.1) V: Various  V03: All other therapeutic products 16 1 1 1 3 1 23 (0.6)  V06: General nutrients 5 1 2 1 1 10 (0.3)  V08: Contrast media 1 1 (0)  Other: general medication indicators 14 2 3 40 73 68 4 44 248 (6.8)  Total, n (%) 2117 (57.7) 20 (0.5) 142 (3.9) 85 (2.3) 439 (12) 161 (4.4) 574 (15.7) 19 (0.5) 109 (3) 3666 (100) *Basger BJ, Moles RJ, Chen TF. Development of an Aggregated System for Classifying Causes of Drug-Related Problems. Ann Pharmacother 2015;49:405–18. ATC, Anatomical Therapeutic Chemical; c-DRPs, causes of drug-related problems; QIs, quality indicators. With regard to the c-DRPs classification system, the most common c-DRPs pertained to ‘drug selection’ (n=2117, 58%), followed by ‘monitoring’ (n=574, 16%) and the ‘drug use process’ (n=439, 12%). The remaining six c-DRPs categories accounted for only 14% of the QIs. Interestingly, only QIs for analgesics (N02) covered all nine c-DRPs categories. In terms of general medication indicators, the largest number of QIs covered ‘Logistics’ (n=73, 29%) among the c-DRPs category, which mainly focus on medication reconciliation problems during transitions of care, such as hospital admission and discharge. A complete list of 2431 QIs is available in online supplementary table S7. 10.1136/bmjopen-2017-020437.supp7Supplementary file 7 Discussion The RUM is important for almost every healthcare setting in every country across the globe. Knowledge of whether medicines are being used in an optimal manner therefore presents a significant international challenge. In this systematic review, we identified 2431 QIs evaluating RUM and classified them using multiple frameworks. The large number of QIs reflects the multidimensional components of RUM and the different perspectives of multidisciplinary stakeholders involved in the RUM. The QI list presented in this review can be used as a comprehensive database and reference for existing content validated QIs pertaining to RUM. All stakeholders involved in quality assurance for RUM, for example, healthcare professionals, researchers and decision makers, can select QIs from the multicategorised QI list for their own purpose. Since healthcare systems and medication guidelines may vary between countries when using the QIs at the local setting, it is important for users to critically review the QIs for their acceptability, feasibility of acquiring necessary data, reliability, sensitivity to change, work load and validity.8 14 The vast majority of the QIs for RUM identified were intended to be used in only a few high-income countries. Low-income and middle-income countries, however, are estimated to have similar rates of medication-related adverse events, and the impact has been reported to be about twice as much in terms of the number of years of healthy life lost.4 Since feasibility of data collection for calculating QI scores in low-income settings remain a concern,151 further efforts for improving the data collection method might need to be made. We found that even though the role of all measurement tools (ie, QIs) relevant to RUM have the goal of quality improvement, the terminology used to describe QIs varied significantly. About 20 name variations were found, which reflects the absence of a universally accepted definition for such tools. For example, Campbell et al8 distinguished QIs from performance indicators, arguing that QIs infer a judgement about the quality of care provided, while performance indicators are statistical devices for monitoring care provided to populations without any necessary inference about quality. However, we found that these terms, ‘quality’ and ‘performance’, were used interchangeably. Hence, further research for standardising the definition that distinguishes these measurement tools is warranted. We also found a significant gap in terms of the problem type (eg, ‘disease-based problems’ (43%), ‘treatment modality-based problems’ (27%) and ‘profession-based problems’ (5%)). Since RUM is facilitated by collaboration in multidisciplinary teams, all healthcare professionals involved in medication treatment should take responsibility for quality assurance, regardless of diseases, care settings and professions. When using Donabedian’s framework, about 94% of the identified QIs related to processes of care. This could be because processes of care are easier to measure, and because process indicators can provide interpretable feedback about care provided.158 In contrast, there was a paucity of outcome indicators. This may be because multiple factors influence health outcomes, many of which are outside the control of individual healthcare professionals. In addition, the difficulty of obtaining sufficient information for assessing outcomes, requiring the linkage of multiple data sources, could be another reason of the limited number of outcome indicators. For outcome indicators to become more useful, multiple confounders such as patient demographic characteristics, and severity of illness, may need to be considered.159 Similarly, there was a low proportion of structural indicators. This may be because they are not sufficiently sensitive for monitoring ongoing performance and they have traditionally been used to monitor standards of healthcare facilities, not RUM.160 It is noteworthy that there is no set requirement for equal proportions of structural, process and outcome indicators in quality measurement. Instead, it is important to recognise the interconnectedness of these measures. For example, high structure indicator scores increase the likelihood of good process indicator scores, which in turn, may lead to higher outcome indicator scores.161 Further research is needed to investigate the associations between the identified QIs in each framework within healthcare settings. We found large differences in the degree to which c-DRPs categories were covered by the identified QIs. Not surprisingly, ‘Drug selection’ accounted for more than half of the QIs, as choosing an inappropriate drug is the main cause of DRPs.3 162 Since focusing on limited c-DRPs categories may divert attention and resources away from other factors contributing to DRPs,163 164 users of QIs should be aware of what c-DRPs categories are not being measured. Like Donabedian’s framework, we do not expect that QIs should be evenly distributed across each of the c-DRPs categories or ATC groups. We do, however, expect that there will be greater QIs in areas of greatest need. These clinical areas may include common areas of practice suspected to be associated with inappropriate use of medicines and significant economic burden (eg, over use of antibiotics for upper respiratory tract infection and overuse of opioid analgesics). Use of QIs in these areas may fill the evidence–practice gaps and minimise subsequent DRPs.165 166 QIs for antithrombotic agents (B01) accounted for the larger proportion of QIs targeting ‘drug selection’, ‘dose selection’, ‘drug use process’ and ‘monitoring’ in c-DRPs categories. This may be explained by the fact that the majority of preventable drug-related admissions have been attributed to antiplatelets and anticoagulants, which have narrow therapeutic indices and high risk of overdose or toxicity,3 and also the fact that medication adherence to long-term antithrombotic therapy remains challenging.167 Likewise, QIs for psychoanaleptics (N06) accounted for the largest part of QIs targeting ‘treatment duration’. Since medication adherence is an ongoing challenge for consumers being treated for depression with antidepressant therapy, it seems appropriate that a relatively large number of QIs have been developed in these categories. In contrast, there were few QIs for some ATC groups, such as dermatological medicines. This has previously been reported in the literature for QIs as a whole, when comparing the scope of dermatology QIs to other medical specialty areas (eg, internal medicine, paediatrics or cardiology).168 This may be because dermatological medicines, especially topical agents, are relatively less harmful and less expensive. Since irrational topical dermatological medication can occur because of drug selection error and patients’ misunderstanding, prescribing, dispensing and administration errors,169 more QIs targeting the wide range of c-DRPs categories may need to be developed for ensuring RUM. Furthermore, when focusing on general medication indicators, QIs largely focused on ‘logistic’ issues such as medication reconciliation at transition points and unavailability of medicines in the c-DRPs category. This differed from medicine class specific QIs, which mainly focused on ‘drug selection’ issues. These differences underscore the importance of the combined use of general medication QIs and medicine class specific QIs for the comprehensive evaluation of RUM. In terms of interpretation of direction of QI scores, we found different methods of scoring: those for evaluating whether necessary or appropriate care was provided and those for evaluating whether unnecessary or inappropriate care was provided. Therefore, care in the interpretation of QI scores is recommended as they have different interpretations based on positively or negatively worded indicators. We also found there were many similar QIs, with only minor differences in wording or definition. These slight differences may be attributed to feasibility of acquiring the data, differences in national guidelines, targeted populations or healthcare systems between locations or countries. However, these minor differences could adversely affect comparability of QI scores and could decrease motivation of healthcare professionals to participate in initiatives if they feel they are being asked the same indicator questions repeatedly. This may be overcome by undertaking a mapping exercise of the QIs identified in our review, with the potential of aggregating some of the QIs. QI is one of the measurement tools to evaluate quality of care at the healthcare facility or group level. QI scores do not directly represent quality of individual patient care but are used as ‘flags’ or ‘alerts’ to potential problems that require further analysis.170 In addition, actions required for quality improvement vary from the level of individual patients, healthcare providers, facilities or healthcare system. Therefore, a multidisciplinary, multilevel quality improvement initiative is needed for comprehensive quality assurance. Strengths and limitations Our review has some notable strengths. This is the first comprehensive review of QIs pertaining to RUM without restriction of disease categories and care settings. In order to do this, a comprehensive literature search was undertaken across multiple databases and websites. Moreover, the classification of QIs was based on multiple frameworks (eg, Donabedian and c-DRPs) for maximum understanding and profiling of the included QIs. The rich dataset of identified QIs can be used as a starting point for healthcare professionals, researchers, decision makers and others, for identifying and selecting existing QIs for the evaluation of RUM. We also identified significant gaps in current quality measurements in each framework, underscoring the need for further QI development in some areas. We do however acknowledge that our approach has some limitations. First, we only included QIs that were developed using consensus methods and excluded QIs if consensus results for QI development were unclear. Therefore, we might have excluded valid indicators during the screening process. Second, although 5% of this review processes were verified by multiple authors, our mapping exercise into the classification system may be viewed as subjective. Third, we identified QIs developed using consensus methods to ensure content validity; however, the methodological rigour of each study was not assessed. Therefore, the quality of the content validity of identified QIs was not reported. Conclusions Overall, by using multiple frameworks, we were able to identify and classify 2431 QIs covering different constructs of RUM. However, this review also pointed to some significant gaps in current quality measurements, making it difficult for healthcare systems to fully assess whether RUM has been achieved or not. The list of the identified QIs can be used as a database for evaluating the achievement of RUM. All stakeholders involved in quality assurance for RUM can select QIs from the multicategorised QI list for their own purpose. In order to more effectively evaluate the extent to which RUM has been achieved, further development and validation of QIs may be required. Supplementary Material Reviewer comments Author's manuscript Contributors: KF developed the review protocol and designed the review questions, carried out database search, articles screening, data extraction and classification and manuscript write up. RJM participated in protocol development, database search, articles screening, data extraction and classification and manuscript review. TFC participated in protocol development, conceptualising the review, designing review questions and database search, article screening, data extraction and classification and manuscript review. Funding: This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. Competing interests: None declared. Patient consent: Not required. Provenance and peer review: Not commissioned; externally peer reviewed. Data sharing statement: Further details on studies included in this review can be retrieved by contacting the corresponding author. ==== Refs References 1. World Health Organization . The pursuit of responsible use of medicines: sharing and learning from country experiences: Geneva: World Health Organization . 2012 http://www.who.int/medicines/publications/responsible_use/en/ (accessed 15 May 2017 ). 2. World Health Organization . Dept. of essential drugs and medicines policy . Promoting rational use of medicines: core components: Geneva: World Health Organization 2002 http://www.who.int/iris/handle/10665/67438 (accessed 5 May 2017 ). 3. Howard RL , Avery AJ , Slavenburg S , et al Which drugs cause preventable admissions to hospital? A systematic review . Br J Clin Pharmacol 2007 ;63 :136 –47 . 10.1111/j.1365-2125.2006.02698.x 16803468 4. World Health Organization . WHO launches global effort to halve medication-related errors in 5 years: Geneva: World Health Organization . 2017 http://www.who.int/mediacentre/news/releases/2017/medication-related-errors/en/ (accessed 5 May 2017 ). 5. Mainz J Defining and classifying clinical indicators for quality improvement . Int J Qual Health Care 2003 ;15 :523 –30 . 10.1093/intqhc/mzg081 14660535 6. Brook RH , McGlynn EA , Shekelle PG Defining and measuring quality of care: a perspective from US researchers . Int J Qual Health Care 2000 ;12 :281 –95 . 10.1093/intqhc/12.4.281 10985266 7. McGlynn E , Asch SM Developing a clinical performance measure . Am J Prev Med 1998 ;14 :14 –21 . 10.1016/S0749-3797(97)00032-9 9566932 8. Campbell SM , et al Improving the quality of health care: Research methods used in developing and applying quality indicators in primary care . BMJ 2003 ;326 :816 –9 . 10.1136/bmj.326.7393.816 12689983 9. Smeulers M , Verweij L , Maaskant JM , et al Quality indicators for safe medication preparation and administration: a systematic review . PLoS One 2015 ;10 :e0122695 10.1371/journal.pone.0122695 25884623 10. Catumbela E , Certal V , Freitas A , et al Definition of a core set of quality indicators for the assessment of HIV/AIDS clinical care: a systematic review . BMC Health Serv Res 2013 ;13 :236 10.1186/1472-6963-13-236 23809537 11. Sadeghi BH , Farhoudi M , Hajebrahimi S , et al A systematic review on clinical indicators, their types and codification processes . Journal of Clinical Research & Governance 2015 ;4 :1 –9 . 12. Copnell B , Hagger V , Wilson SG , et al Measuring the quality of hospital care: an inventory of indicators . Intern Med J 2009 ;39 :352 –60 . 10.1111/j.1445-5994.2009.01961.x 19323697 13. Moher D , Liberati A , Tetzlaff J , et al PRISMA Group . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Ann Intern Med 2009 ;151 :264 –9 . 10.7326/0003-4819-151-4-200908180-00135 19622511 14. Campbell SM , Braspenning J , Hutchinson A , et al Research methods used in developing and applying quality indicators in primary care . Qual Saf Health Care 2002 ;11 :358 –64 . 10.1136/qhc.11.4.358 12468698 15. Klassen A , Miller A , Anderson N , et al Performance measurement and improvement frameworks in health, education and social services systems: a systematic review . International Journal for Quality in Health Care 2010 ;22 :44 –69 . 10.1093/intqhc/mzp057 19951964 16. Haller G Quality and safety indicators in anesthesia: a systematic review . Anesthesiology 2009 ;110 :1158 –75 .19352148 17. Alessandrini E , Varadarajan K , Alpern ER , et al Pediatric Emergency Care Applied Research Network . Emergency department quality: an analysis of existing pediatric measures . Acad Emerg Med 2011 ;18 :519 –26 . 10.1111/j.1553-2712.2011.01057.x 21569170 18. Evans SM , Lowinger JS , Sprivulis PC , et al Prioritizing quality indicator development across the healthcare system: identifying what to measure . Intern Med J 2009 ;39 :648 –54 . 10.1111/j.1445-5994.2008.01733.x 19371394 19. Donabedian A The definition of quality and approaches to its assessment . Ann Arbor, Mich : Health Administration Press , 1980 . 20. Sun H , Liu M , Hou S Quality indicators for acute myocardial infarction care in China . Int J Qual Health Care 2011 ;23 :365 –74 . 10.1093/intqhc/mzr020 21561980 21. Tropea J , Amatya B , Brand CA Use of consensus methods to select clinical indicators to assess activities to minimise functional decline among older hospitalised patients . Aust Health Rev 2011 ;35 :404 –11 .22126941 22. World Health Organization . The Anatomical Therapeutic Chemical (ATC) classification system: structure and principles . Geneva : World Health Organization , 2011 https://www.whocc.no/atc/structure_and_principles/ (accessed 5 May 2017 ). 23. Teichert M , Schoenmakers T , Kylstra N , et al Quality indicators for pharmaceutical care: a comprehensive set with national scores for Dutch community pharmacies . Int J Clin Pharm 2016 ;38 :870 –9 . 10.1007/s11096-016-0301-x 27107583 24. Basger BJ , Moles RJ , Chen TF Development of an Aggregated System for Classifying Causes of Drug-Related Problems . Ann Pharmacother 2015 ;49 :405 –18 . 10.1177/1060028014568008 25614526 25. Hadorn DC , Baker DW , Kamberg CJ , et al Phase II of the AHCPR-sponsored heart failure guideline: translating practice recommendations into review criteria . Jt Comm J Qual Improv 1996 ;22 :265 –76 . 10.1016/S1070-3241(16)30230-9 8743063 26. Asch SM , Kerr EA , Lapuerta P , et al A new approach for measuring quality of care for women with hypertension . Arch Intern Med 2001 ;161 :1329 –35 . 10.1001/archinte.161.10.1329 11371262 27. Mikuls TR , MacLean CH , Olivieri J , et al Quality of care indicators for gout management . Arthritis & Rheumatism 2004 ;50 :937 –43 . 10.1002/art.20102 15022337 28. Saliba D , Solomon D , Rubenstein L , et al Quality Indicators for the Management of Medical Conditions in Nursing Home Residents . J Am Med Dir Assoc 2004 ;5 :297 –309 . 10.1016/S1525-8610(04)70019-5 15357887 29. Krumholz HM , Anderson JL , Brooks NH , et al ACC/AHA clinical performance measures for adults with ST-elevation and non-ST-elevation myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures (Writing Committee to Develop Performance Measures on ST-Elevation and Non-ST-Elevation Myocardial Infarction) . Circulation 2006 ;113 :732 –61 . 10.1161/CIRCULATIONAHA.106.172860 16391153 30. McGory ML , Shekelle PG , Ko CY Development of quality indicators for patients undergoing colorectal cancer surgery . J Natl Cancer Inst 2006 ;98 :1623 –33 . 10.1093/jnci/djj438 17105985 31. Mularski RA , Curtis JR , Billings JA , et al Proposed quality measures for palliative care in the critically ill: a consensus from the robert wood johnson foundation critical care workgroup . Crit Care Med 2006 ;34 :S404 –S411 . 10.1097/01.CCM.0000242910.00801.53 17057606 32. Mangione-Smith R , DeCristofaro AH , Setodji CM , et al The Quality of ambulatory care delivered to children in the United States . N Engl J Med Overseas Ed 2007 ;357 :1515 –23 . 10.1056/NEJMsa064637 33. Smith KL , Soriano TA , Boal J Brief communication: National quality-of-care standards in home-based primary care . Ann Intern Med 2007 ;146 :188 –92 . 10.7326/0003-4819-146-3-200702060-00008 17283350 34. Wenger NS , Roth CP , Shekelle P , et al Introduction to the assessing care of vulnerable elders-3 quality indicator measurement set . J Am Geriatr Soc 2007 ;55 Suppl 2 :S247 –S252 . 10.1111/j.1532-5415.2007.01328.x 17910544 35. Estes NA , Halperin JL , Calkins H , et al ACC/AHA/Physician Consortium 2008 clinical performance measures for adults with nonvalvular atrial fibrillation or atrial flutter: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and the Physician Consortium for Performance Improvement (Writing Committee to Develop Clinical Performance Measures for Atrial Fibrillation): developed in collaboration with the Heart Rhythm Society . Circulation 2008 ;117 :1101 –20 . 10.1161/CIRCULATIONAHA.107.187192 18283199 36. Bilimoria KY , Bentrem DJ , Lillemoe KD , et al Assessment of pancreatic cancer care in the United States based on formally developed quality indicators . J Natl Cancer Inst 2009 ;101 :848 –59 . 10.1093/jnci/djp107 19509366 37. Lorenz KA , Dy SM , Naeim A , et al Quality Measures for Supportive Cancer Care: The Cancer Quality-ASSIST Project . J Pain Symptom Manage 2009 ;37 :943 –64 . 10.1016/j.jpainsymman.2008.05.018 19359135 38. McGory ML , Kao KK , Shekelle PG , et al Developing quality indicators for elderly surgical patients . Ann Surg 2009 ;250 :338 –47 . 10.1097/SLA.0b013e3181ae575a 19638913 39. Yazdany J , Panopalis P , Gillis JZ , et al A quality indicator set for systemic lupus erythematosus . Arthritis & Rheumatism 2009 ;61 :370 –7 . 10.1002/art.24356 19248127 40. Cheng EM , Crandall CJ , Bever CT , et al Quality indicators for multiple sclerosis . Mult Scler 2010 ;16 :970 –80 . 10.1177/1352458510372394 20562162 41. Kanwal F , Barkun A , Gralnek IM , et al Measuring quality of care in patients with nonvariceal upper gastrointestinal hemorrhage: development of an explicit quality indicator set . Am J Gastroenterol 2010 ;105 :1710 –8 . 10.1038/ajg.2010.180 20686458 42. Kanwal F , Kramer J , Asch SM , et al An explicit quality indicator set for measurement of quality of care in patients with cirrhosis . Clin Gastroenterol Hepatol 2010 ;8 :709 –17 . 10.1016/j.cgh.2010.03.028 20385251 43. Schenck AP , Rokoske FS , Durham DD , et al The PEACE Project: identification of quality measures for hospice and palliative care . J Palliat Med 2010 ;13 :1451 –9 . 10.1089/jpm.2010.0238 21155640 44. Khanna D , Kowal-Bielecka O , Khanna PP , et al Quality indicator set for systemic sclerosis . Clinical and Experimental Rheumatology 2011 ;29 :S33 –9 .21586216 45. SooHoo NF , Lieberman JR , Farng E , et al Development of quality of care indicators for patients undergoing total hip or total knee replacement . BMJ Qual Saf 2011 ;20 :153 –7 . 10.1136/bmjqs.2009.032524 46. Wang CJ , Kavanagh PL , Little AA , et al Quality-of-care indicators for children with sickle cell disease . Pediatrics 2011 ;128 :484 –93 . 10.1542/peds.2010-1791 21844055 47. Anger JT , Scott VC , Kiyosaki K , et al Development of quality indicators for women with urinary incontinence . Neurourol Urodyn 2013 ;32 :1058 –63 . 10.1002/nau.22353 24105879 48. Jackson GL , Zullig LL , Zafar SY , et al Using NCCN clinical practice guidelines in oncology to measure the quality of colorectal cancer care in the veterans health administration . J Natl Compr Canc Netw 2013 ;11 :431 –41 . 10.6004/jnccn.2013.0058 23584346 49. Melmed GY , Siegel CA , Spiegel BM , et al Quality indicators for inflammatory bowel disease: development of process and outcome measures. Erratum appears in Inflamm Bowel Dis . Inflamm Bowel Dis 2013 ;19 :662 –8 .23388547 50. Wang CJ , Jonas R , Fu CM , et al Quality-of-care indicators for infantile spasms . J Child Neurol 2013 ;28 :13 –20 . 10.1177/0883073812443590 22566712 51. Yadlapati R , Gawron AJ , Bilimoria K , et al Development of quality measures for the care of patients with gastroesophageal reflux disease . Clin Gastroenterol Hepatol 2015 ;13 :874 –83 . 10.1016/j.cgh.2014.11.012 25460560 52. Faro EZ , Wang CJ , Oyeku SO Quality indicator development for positive screen follow-up for sickle cell disease and trait . Am J Prev Med 2016 ;51 :S48 –S54 . 10.1016/j.amepre.2016.01.005 27320465 53. Vila PM , Lieu JEC , Hullar TE , et al Developing quality measures for adult cochlear implant centers: preliminary findings . Otolaryngology-Head & Neck Surgery 2016 ;155 :748 –52 .27554517 54. Yazdany J , Robbins M , Schmajuk G , et al Development of the american college of rheumatology’s rheumatoid arthritis electronic clinical quality measures . Arthritis Care Res 2016 ;68 :1579 –90 . 10.1002/acr.22984 55. Chowdhury D , Gurvitz M , Marelli A , et al Development of quality metrics in ambulatory pediatric cardiology . J Am Coll Cardiol 2017 ;69 :541 –55 . 10.1016/j.jacc.2016.11.043 28153110 56. Hepner KA , Watkins KE , Farmer CM , et al Quality of care measures for the management of unhealthy alcohol use . J Subst Abuse Treat 2017 ;76 :11 –17 . 10.1016/j.jsat.2016.11.006 28340902 57. Ingraham A , Nathens A , Peitzman A , et al Assessment of emergency general surgery care based on formally developed quality indicators . Surgery 2017 ;162 :397 –407 . 10.1016/j.surg.2017.03.025 28647046 58. Mangione-Smith R , Roth CP , Britto MT , et al Development and Testing of the Pediatric Respiratory Illness Measurement System (PRIMES) Quality Indicators . Hosp Pediatr 2017 ;7 :125 –33 . 10.1542/hpeds.2016-0182 28223319 59. Odetola FO , Freed G , Shevrin C , et al In-hospital quality-of-care measures for pediatric sepsis syndrome . Pediatrics 2017 ;140 :e20170350 –8 . 10.1542/peds.2017-0350 60. Parast L , Bardach NS , Burkhart Q , et al Development of new quality measures for hospital-based care of suicidal youth . Acad Pediatr 2018 ;18 :248 –55 . 10.1016/j.acap.2017.09.017 29100860 61. Mackinnon NJ , Hepler CD Preventable drug-related morbidity in older adults 1. Indicator development . J Manag Care Pharm 2002 ;8 :365 –71 .14613403 62. Robertson HA , MacKinnon NJ Development of a list of consensus-approved clinical indicators of preventable drug-related morbidity in older adults . Clin Ther 2002 ;24 :1595 –613 . 10.1016/S0149-2918(02)80063-7 12462289 63. Burge FI , Bower K , Putnam W , et al Quality indicators for cardiovascular primary care . Can J Cardiol 2007 ;23 :383 –8 . 10.1016/S0828-282X(07)70772-9 17440644 64. Kröger E , Tourigny A , Morin D , et al Selecting process quality indicators for the integrated care of vulnerable older adults affected by cognitive impairment or dementia . BMC Health Serv Res 2007 ;7 :195 10.1186/1472-6963-7-195 18047668 65. Dt K , Wijeysundera HC , Zhu X , et al Canadian quality indicators for percutaneous coronary interventions . Can J Cardiol 2008 ;24 :899 –903 .19052669 66. MacKinnon NJ , Hartnell NR , Black EK , et al Development of clinical indicators for type 2 diabetes . Canadian Pharmacists Journal 2008 ;141 :120 –8 .doi:10.3821/1913-701X(2008)141[120:DOCIFT]2.0.CO;2 67. Nigam R , Mackinnon NJ , U D , et al Development of canadian safety indicators for medication use . Healthc Q 2008 ;11 :47 –53 . 10.12927/hcq.2008.19649 18382161 68. Jv T , Khalid L , Donovan LR , et al Indicators of quality of care for patients with acute myocardial infarction . CMAJ: Canadian Medical Association Journal 2008 ;179 :909 –15 .18936456 69. Dixon E , Armstrong C , Maddern G , et al Development of Quality Indicators of Care for Patients Undergoing Hepatic Resection for Metastatic Colorectal Cancer Using a Delphi Process . J Surg Res 2009 ;156 :32 –8 . 10.1016/j.jss.2009.03.084 19592015 70. Teresato GA , Lougheed MD , et al Evidence-based performance indicators of primary care for asthma: a modified RAND Appropriateness Method . Int J Qual Health Care 2010 ;22 :476 –85 .20978002 71. Krzyzanowska MK , Barbera L , Elit L , et al Identifying population-level indicators to measure the quality of cancer care for women . Int J Qual Health Care 2011 ;23 :554 –64 . 10.1093/intqhc/mzr043 21791577 72. Schull MJ , Guttmann A , Leaver CA , et al Prioritizing performance measurement for emergency department care: consensus on evidencebased quality of care indicators . CJEM 2011 ;13 :300 –9 . 10.2310/8000.2011.110334 21955411 73. Addington DE , Mckenzie E , Wang J , et al Development of a core set of performance measures for evaluating schizophrenia treatment services . Psychiatr Serv 2012 ;63 :584 –91 . 10.1176/appi.ps.201100453 22476226 74. Stang AS , Straus SE , Crotts J , et al Quality indicators for high acuity pediatric conditions . Pediatrics 2013 ;132 :752 –62 . 10.1542/peds.2013-0854 24062374 75. Darling G , Malthaner R , Dickie J , et al Quality indicators for non-small cell lung cancer operations with use of a modified Delphi consensus process . Ann Thorac Surg 2014 ;98 :183 –90 . 10.1016/j.athoracsur.2014.03.001 24775804 76. Nguyen GC , Devlin SM , Afif W , et al Defining quality indicators for best-practice management of inflammatory bowel disease in Canada . Can J Gastroenterol Hepatol 2014 ;28 :275 –85 . 10.1155/2014/941245 24839622 77. Santana MJ , Stelfox HT Trauma Quality Indicator Consensus Panel . Development and evaluation of evidence-informed quality indicators for adult injury care . Ann Surg 2014 ;259 :186 –92 . 10.1097/SLA.0b013e31828df98e 23657078 78. Barber CE , Patel JN , Woodhouse L , et al Development of key performance indicators to evaluate centralized intake for patients with osteoarthritis and rheumatoid arthritis . Arthritis Res Ther 2015 ;17 :322 10.1186/s13075-015-0843-7 26568556 79. Fernandes O , Gorman SK , Slavik RS , et al Development of clinical pharmacy key performance indicators for hospital pharmacists using a modified Delphi approach . Ann Pharmacother 2015 ;49 :656 –69 . 10.1177/1060028015577445 25780250 80. Khare SR , Batist G , Bartlett G Identification of performance indicators across a network of clinical cancer programs . Curr Oncol 2016 ;23 :81 –90 . 10.3747/co.23.2789 27122972 81. McKelvie RS , Heckman GA , Blais C , et al Canadian Cardiovascular Society Quality Indicators for Heart Failure . Can J Cardiol 2016 ;32 :1038.e5 –1038.e9 . 10.1016/j.cjca.2015.12.027 82. Khare SR , Aprikian A , Black P , et al Quality indicators in the management of bladder cancer: A modified Delphi study . Urol Oncol 2017 ;35 :328 –34 . 10.1016/j.urolonc.2016.12.003 28065393 83. Tu JV , Maclagan LC , Ko DT , et al The Cardiovascular Health in Ambulatory Care Research Team performance indicators for the primary prevention of cardiovascular disease: a modified Delphi panel study . CMAJ open 2017 ;5 :E315 –E321 . 84. Tu K , Bevan L , Hunter K , et al Quality indicators for the detection and management of chronic kidney disease in primary care in Canada derived from a modified Delphi panel approach . CMAJ Open 2017 ;5 :E74 –E81 . 10.9778/cmajo.20160113 85. Chartrand M , Guénette L , Brouillette D , et al Development of Quality Indicators to Assess Oral Anticoagulant Management in Community Pharmacies for Patients with Atrial Fibrillation . J Manag Care Spec Pharm 2018 ;24 :357 –65 . 10.18553/jmcp.2018.24.4.357 29578847 86. Mukerji G , Halperin I , Hunter K , et al Developing a set of indicators to monitor quality in ambulatory diabetes care using a modified Delphi panel process . International Journal for Quality in Health Care 2018 ;30 :65 –74 . 10.1093/intqhc/mzx167 29340632 87. Muijrers PEM , Janknegt R , Sijbrandij J , et al Prescribing indicators: Development and validation of guideline-based prescribing indicators as an instrument to measure the variation in the prescribing behaviour of general practitioners . Eur J Clin Pharmacol 2004 ;60 :739 –46 .15517226 88. Mourad SM , Hermens RPMG , Nelen WLDM , et al Guideline-based development of quality indicators for subfertility care . Hum Reprod 2007 ;22 :2665 –72 . 10.1093/humrep/dem215 17664242 89. Drašković I , Vernooij-Dassen M , Verhey F , et al Development of quality indicators for memory clinics . Int J Geriatr Psychiatry 2008 ;23 :119 –28 . 10.1002/gps.1848 17582827 90. Martirosyan L , Braspenning J , Denig P , et al Prescribing quality indicators of type 2 diabetes mellitus ambulatory care . Quality and Safety in Health Care 2008 ;17 :318 –23 . 10.1136/qshc.2007.024224 18842968 91. van der Ploeg E , Depla MFIA , Shekelle P , et al Developing quality indicators for general practice care for vulnerable elders; transfer from US to The Netherlands . Quality and Safety in Health Care 2008 ;17 :291 –5 . 10.1136/qshc.2007.023226 18678728 92. Perry M , Drasković I , van Achterberg T , et al Development and validation of quality indicators for dementia diagnosis and management in a primary care setting . J Am Geriatr Soc 2010 ;58 :557 –63 . 10.1111/j.1532-5415.2010.02726.x 20210818 93. Stienen JJC , Tabbers MM , Benninga MA , et al Development of quality indicators based on a multidisciplinary, evidence-based guideline on pediatric constipation . Eur J Pediatr 2011 ;170 :1513 –9 . 10.1007/s00431-011-1458-x 21487681 94. Wierenga PC , Klopotowska JE , Smorenburg SM , et al Quality Indicators for In-Hospital Pharmaceutical Care of Dutch Elderly Patients . Drugs Aging 2011 ;28 :295 –304 . 10.2165/11587700-000000000-00000 21428464 95. Luitjes SH , Wouters MG , Franx A , et al Guideline-based development of quality indicators for hypertensive diseases in pregnancy . Hypertens Pregnancy 2013 ;32 :20 –31 . 10.3109/10641955.2012.697950 22958211 96. van den Bosch CMA , Hulscher MEJL , Natsch S , et al Development of quality indicators for antimicrobial treatment in adults with sepsis . BMC Infect Dis 2014 ;14 :345 10.1186/1471-2334-14-345 24950718 97. van den Bosch CM , Geerlings SE , Natsch S , et al Quality indicators to measure appropriate antibiotic use in hospitalized adults . Clin Infect Dis 2015 ;60 :281 –91 . 10.1093/cid/ciu747 25266285 98. Woiski MD , Scheepers HC , Liefers J , et al Guideline-based development of quality indicators for prevention and management of postpartum hemorrhage . Acta Obstet Gynecol Scand 2015 ;94 :1118 –27 . 10.1111/aogs.12718 26222391 99. Hommel I , van Gurp PJ , Tack CJ , et al Perioperative diabetes care: development and validation of quality indicators throughout the entire hospital care pathway . BMJ Qual Saf 2016 ;25 :525 –34 . 10.1136/bmjqs-2015-004112 100. Smits KPJ , Sidorenkov G , Bilo HJG , et al Development and initial validation of prescribing quality indicators for patients with chronic kidney disease . Nephrology Dialysis Transplantation 2016 ;31 :1876 –86 . 10.1093/ndt/gfv420 101. Smits KP , Sidorenkov G , Kleefstra N , et al Development and validation of prescribing quality indicators for patients with type 2 diabetes . Int J Clin Pract 2017 ;71 :e12922 10.1111/ijcp.12922 102. Broccoli MC , Moresky R , Dixon J , et al Defining quality indicators for emergency care delivery: findings of an expert consensus process by emergency care practitioners in Africa . BMJ Glob Health 2018 ;3 :e000479 10.1136/bmjgh-2017-000479 103. Caughey GE , Kalisch Ellett LM , Wong TY Development of evidence-based Australian medication-related indicators of potentially preventable hospitalisations: a modified RAND appropriateness method . BMJ Open 2014 ;4 :e004625 10.1136/bmjopen-2013-004625 104. Nag N , Millar J , Davis ID , et al Development of Indicators to Assess Quality of Care for Prostate Cancer . Eur Urol Focus 2018 ;4 :57 –63 . 10.1016/j.euf.2016.01.016 28753751 105. O’Connor DW , D’Cunha C , Clifton T , et al Quality indicators of psychotropic prescribing to people with dementia in aged psychiatry inpatient units . Aging Ment Health 2017 :1 –6 . 10.1080/13607863.2017.1366418 106. Sibthorpe B , Agostino J , Coates H , et al Indicators for continuous quality improvement for otitis media in primary health care for Aboriginal and Torres Strait Islander children . Aust J Prim Health 2017 ;23 :1 –9 . 10.1071/PY16096 28088980 107. Stordeur S , Vrijens F , Devriese S , et al Developing and measuring a set of process and outcome indicators for breast cancer . Breast 2012 ;21 :253 –60 . 10.1016/j.breast.2011.10.003 22056787 108. Grypdonck L , Aertgeerts B , Luyten F , et al Development of quality indicators for an integrated approach of knee osteoarthritis . J Rheumatol 2014 ;41 :1155 –62 . 10.3899/jrheum.130680 24737907 109. Stordeur S , Vlayen J , Vrijens F , et al Quality indicators for oesophageal and gastric cancer: a population-based study in Belgium, 2004-2008 . Eur J Cancer Care 2015 ;24 :376 –86 . 10.1111/ecc.12279 110. De Schreye R , Houttekier D , Deliens L , et al Developing indicators of appropriate and inappropriate end-of-life care in people with Alzheimer’s disease, cancer or chronic obstructive pulmonary disease for population-level administrative databases: A RAND/UCLA appropriateness study . Palliat Med 2017 ;31 :932 –45 . 10.1177/0269216317705099 28429629 111. Leemans K , Deliens L , Van den Block L , et al Systematic quality monitoring for specialized palliative care services: development of a minimal set of quality indicators for palliative care study (QPAC) . Am J Hosp Palliat Care 2017 ;34 :532 –46 . 10.1177/1049909116642174 27059060 112. de Carvalho AGR , de Moraes APP , Tanaka LMS , et al Quality in intensive care units: proposal of an assessment instrument . BMC Res Notes 2017 ;10 :222 10.1186/s13104-017-2563-3 28651640 113. Bao H , Yang F , Wang X , et al Developing a set of quality indicators for breast cancer care in China . Int J Qual Health Care 2015 ;27 :291 –6 . 10.1093/intqhc/mzv042 26104388 114. Chen L , Huang LH , Xing MY , et al Using the Delphi method to develop nursing-sensitive quality indicators for the NICU . J Clin Nurs 2017 ;26 :502 –13 . 10.1111/jocn.13474 27404730 115. Wu Q , Huang LH , Xing MY , et al Establishing nursing-sensitive quality indicators for the operating room: A cross-sectional Delphi survey conducted in China . Aust Crit Care 2017 ;30 :44 –52 . 10.1016/j.aucc.2016.04.003 27160417 116. Li W , Zeng L , Li J , et al Development of indicators for assessing rational drug use to treat community-acquired pneumonia in children in hospitals and clinics: A modified Delphi study . Medicine 2017 ;96 :e9308 10.1097/MD.0000000000009308 29390500 117. Wang X , Su S , Li S , et al Development of quality indicators for non-small cell lung cancer care: a first step toward assessing and improving quality of cancer care in China . BMC Cancer 2017 ;17 :603 10.1186/s12885-017-3602-0 28859609 118. Ju QY , Huang LH , Zhao XH , et al Development of evidence-based nursing-sensitive quality indicators for emergency nursing: a delphi study . J Clin Nurs 2018 10.1111/jocn.14256 119. Tang X , Chen X , Pang Y , et al The development of quality indicators for home care in China . Int J Qual Health Care 2018 ;30 :208 –18 . 10.1093/intqhc/mzx202 29385451 120. Saust LT , Bjerrum L , Arpi M , et al Quality indicators for the diagnosis and antibiotic treatment of acute respiratory tract infections in general practice: a RAND Appropriateness Method . Scand J Prim Health Care 2017 ;35 :192 –200 . 10.1080/02813432.2017.1333305 28574299 121. Boulkedid R , Sibony O , Goffinet F , et al Quality indicators for continuous monitoring to improve maternal and infant health in maternity departments: a modified Delphi survey of an international multidisciplinary panel . PLoS One 2013 ;8 :e60663 10.1371/journal.pone.0060663 23577143 122. Follmann M , Schadendorf D , Kochs C , et al Quality assurance for care of melanoma patients based on guideline-derived quality indicators and certification . J Dtsch Dermatol Ges 2014 ;12 :139 –47 . 10.1111/ddg.12238 24238575 123. Hussein RJ , Krohn R , Kaufmann-Kolle P , et al Quality indicators for the use of systemic antibiotics in dentistry . Z Evid Fortbild Qual Gesundhwes 2017 ;122 :1 –8 . 10.1016/j.zefq.2017.04.007 28522282 124. Wakai A , O’Sullivan R , Staunton P , et al Development of key performance indicators for emergency departments in Ireland using an electronic modified-Delphi consensus approach . Eur J Emerg Med 2013 ;20 :109 –14 . 10.1097/MEJ.0b013e328351e5d8 22382650 125. Murphy A , Wakai A , Walsh C , et al Development of key performance indicators for prehospital emergency care . Emerg Med J 2016 ;33 :286 –92 . 10.1136/emermed-2015-204793 26796739 126. Barry E , O’Brien K , Moriarty F , et al PIPc study: development of indicators of potentially inappropriate prescribing in children (PIPc) in primary care using a modified Delphi technique . BMJ Open 2016 ;6 :e012079 10.1136/bmjopen-2016-012079 127. Fukuma S , Shimizu S , Niihata K , et al Development of quality indicators for care of chronic kidney disease in the primary care setting using electronic health data: a RAND-modified Delphi method . Clin Exp Nephrol 2017 ;21 :247 –56 . 10.1007/s10157-016-1274-8 27145768 128. Masaki H , Kawai N , Matsumoto K , et al Consensus development of quality indicators for end-of-life care for elders in Japan . Int J Nurs Pract 2017 ;23 Suppl 1 :e12562 10.1111/ijn.12562 129. Ueda K , Ohtera S , Kaso M , et al Development of quality indicators for low-risk labor care provided by midwives using a RAND-modified Delphi method . BMC Pregnancy Childbirth 2017 ;17 :315 10.1186/s12884-017-1468-4 28938879 130. Pérez-Cuevas R , Doubova SV , Suarez-Ortega M , et al Evaluating quality of care for patients with type 2 diabetes using electronic health record information in Mexico . BMC Med Inform Decis Mak 2012 ;12 :50 10.1186/1472-6947-12-50 22672471 131. Doubova SV , Pérez-Cuevas R , Ortiz-Panozo E , et al Evaluation of the quality of antenatal care using electronic health record information in family medicine clinics of Mexico City . BMC Pregnancy Childbirth 2014 ;14 :168 10.1186/1471-2393-14-168 24885103 132. Petek D , Campbell S , Serec M , et al Quality indicators of cardiovascular disease prevention for primary care in Slovenia . Zdravniski Vestnik 2012 ;81 :687 –98 . 133. Minaya-Muñoz F , Medina-Mirapeix F , Valera-Garrido F Quality measures for the care of patients with lateral epicondylalgia . BMC Musculoskelet Disord 2013 ;14 :310 10.1186/1471-2474-14-310 24172311 134. Calvet X , Panés J , Alfaro N , et al Delphi consensus statement: quality indicators for inflammatory bowel disease comprehensive care units . J Crohns Colitis 2014 ;8 :240 –51 . 10.1016/j.crohns.2013.10.010 24295646 135. Ruiz-Canela-Cáceres J , Aquino-Llinares N , Sánchez-Diaz JM , et al Indicators for childhood asthma in Spain, using the Rand method . Allergol Immunopathol 2015 ;43 :147 –56 . 10.1016/j.aller.2013.12.005 136. Soria-Aledo V , Angel-Garcia D , Martinez-Nicolas I , et al Development and pilot study of an essential set of indicators for general surgery services . Cir Esp 2016 ;94 :502 –10 . 10.1016/j.ciresp.2016.06.009 27499298 137. Bianchi V , Spitale A , Ortelli L , et al Quality indicators of clinical cancer care (QC3) in colorectal cancer . BMJ Open 2013 ;3 :e002818 10.1136/bmjopen-2013-002818 138. Chung KP , Lai MS , Cheng SH , et al Organization-based performance measures of cancer care quality: core measure development for breast cancer in Taiwan . Eur J Cancer Care 2008 ;17 :5 –18 . 10.1111/j.1365-2354.2007.00796.x 139. Chung KP , Chang YJ , Lai MS , et al Is quality of colorectal cancer care good enough? Core measures development and its application for comparing hospitals in Taiwan . BMC Health Serv Res 2010 ;10 :27 10.1186/1472-6963-10-27 20105287 140. Cantrill JA , Sibbald B , Buetow S Indicators of the appropriateness of long-term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility, and reliability . Qual Health Care 1998 ;7 :130 –5 . 10.1136/qshc.7.3.130 10185138 141. Morris CJ , Cantrill JA Preventing drug-related morbidity--the development of quality indicators . J Clin Pharm Ther 2003 ;28 :295 –305 . 10.1046/j.1365-2710.2003.00496.x 12911682 142. Steel N , Melzer D , Shekelle PG , et al Developing quality indicators for older adults: transfer from the USA to the UK is feasible . Qual Saf Health Care 2004 ;13 :260 –4 . 10.1136/qshc.2004.010280 15289628 143. Tully MP , Javed N , Cantrill JA Development and face validity of explicit indicators of appropriateness of long term prescribing . Pharm World Sci 2005 ;27 :407 –13 . 10.1007/s11096-005-0340-1 16341749 144. Gill PJ , O’Neill B , Rose P , et al Primary care quality indicators for children: measuring quality in UK general practice . Br J Gen Pract 2014 ;64 :e752 –e757 . 10.3399/bjgp14X682813 25452539 145. Spencer R , Bell B , Avery AJ , et al Identification of an updated set of prescribing--safety indicators for GPs . Br J Gen Pract 2014 ;64 :e181 –e190 . 10.3399/bjgp14X677806 24686882 146. Campbell SM , Ludt S , Van Lieshout J , et al Quality indicators for the prevention and management of cardiovascular disease in primary care in nine European countries . Eur J Cardiovasc Prev Rehabil 2008 ;15 :509 –15 . 10.1097/HJR.0b013e328302f44d 18695594 147. Adriaenssens N , Coenen S , Tonkin-Crine S , et al European Surveillance of Antimicrobial Consumption (ESAC): disease-specific quality indicators for outpatient antibiotic prescribing . BMJ Qual Saf 2011 ;20 :764 –72 . 10.1136/bmjqs.2010.049049 148. Petersson IF , Strömbeck B , Andersen L , et al Development of healthcare quality indicators for rheumatoid arthritis in Europe: the eumusc.net project . Ann Rheum Dis 2014 ;73 :906 –8 . 10.1136/annrheumdis-2013-203765 23960093 149. Hermann RC , Mattke S , Somekh D , et al Quality indicators for international benchmarking of mental health care . Int J Qual Health Care 2006 ;18 Suppl 1 :31 –8 . 10.1093/intqhc/mzl025 16954514 150. Barber CE , Marshall DA , Alvarez N , et al Development of Cardiovascular Quality Indicators for Rheumatoid Arthritis: Results from an International Expert Panel Using a Novel Online Process . J Rheumatol 2015 ;42 :1548 –55 . 10.3899/jrheum.141603 26178275 151. Ntoburi S , Hutchings A , Sanderson C , et al Development of paediatric quality of inpatient care indicators for low-income countries - A Delphi study . BMC Pediatr 2010 ;10 :90 10.1186/1471-2431-10-90 21144065 152. Idänpään-Heikkilä UM , Lambie L , Mattke S , et al Selecting indicators for the quality of cardiac care at the health system level in Organization for Economic Co-operation and Development countries . Int J Qual Health Care 2006 ;18 Suppl 1 :39 –44 . 10.1093/intqhc/mzl028 16954515 153. Australian Commission on Safety and Quality in Health Care. Practice-level indicators of safety and quality for primary health care specification: Australian Commission on Safety and Quality in Health Care . 2012 https://www.safetyandquality.gov.au/wp-content/uploads/2012/02/Practice-level-indicators-for-primary-health-care-specification-V1.0-October-2012.pdf (accessed 5 May 2017 ). 154. Australian Commission on Safety and Quality in Health Care and NSW Therapeutic Advisory Group Inc . National Quality use of medicines indicators for australian hospitals Sydney: ACSQHC . 2014 https://www.safetyandquality.gov.au/wp-content/uploads/2014/11/SAQ127_National_QUM_Indicators_V14-FINAL-D14-39602.pdf (accessed 5 May 2017 ). 155. the Department of Health & Human Services, State Government of Victoria, Australia . Quality indicators in PSRACS resource materials 2015 . 2015 https://www2.health.vic.gov.au/about/publications/policiesandguidelines/quality-indicators-psracs-resource-materials-2015 (accessed 9 April 2018 ). 156. The NICE menu of general practice and clinical commissioning group indicators: The National Institute for Health and Care Excellence . 2016 https://www.nice.org.uk/Media/Default/Standards-and-indicators/indicator-menu-update-aug-16.pdf (accessed 5 May 2017 ). 157. AfHRa Q National Quality Measures Clearinghouse: U . S. Department of Health & Human Services 2017 https://www.qualitymeasures.ahrq.gov/ (accessed 5 April 2018 ). 158. Rubin HR , Pronovost P , Diette GB The advantages and disadvantages of process-based measures of health care quality . Int J Qual Health Care 2001 ;13 :469 –74 . 10.1093/intqhc/13.6.469 11769749 159. Iezzoni LI The risks of risk adjustment . JAMA 1997 ;278 :1600 –7 . 10.1001/jama.1997.03550190064046 9370507 160. Nau DP Measuring pharmacy quality . J Am Pharm Assoc 2009 ;49 :154 –63 . 10.1331/JAPhA.2009.09019 161. Gardner K , Mazza D Quality in general practice - definitions and frameworks . Aust Fam Physician 2012 ;41 :151 –4 .22396930 162. Silva C , Ramalho C , Luz I , et al Drug-related problems in institutionalized, polymedicated elderly patients: opportunities for pharmacist intervention . Int J Clin Pharm 2015 ;37 :327 –34 . 10.1007/s11096-014-0063-2 25637404 163. Elliot-Smith A , Morgan MA , Elliot S How do we compare? Applying UK pay for performance indicators to an Australian general practice . Aust Fam Physician 2010 ;39 :43 .20369134 164. Hannon KL , Lester HE , Campbell SM Patients' views of pay for performance in primary care: a qualitative study . Br J Gen Pract 2012 ;62 :322 –8 . 10.3399/bjgp12X641438 165. Wollersheim H , Hermens R , Hulscher M , et al Clinical indicators: development and applications . Neth J Med 2007 ;65 :15 –22 .17293635 166. Mainz J Developing evidence-based clinical indicators: a state of the art methods primer . Int J Qual Health Care 2003 ;15 Suppl 1 :5i –11 . 10.1093/intqhc/mzg084 12630795 167. Kneeland PP , Fang MC Current issues in patient adherence and persistence: focus on anticoagulants for the treatment and prevention of thromboembolism . Patient Prefer Adherence 2010 ;4 :51 –60 .20361065 168. Freedman JD , Gottlieb AB , Lizzul PF Physician performance measurement: tiered networks and dermatology (an opportunity and a challenge) . J Am Acad Dermatol 2011 ;64 :1164 –9 . 10.1016/j.jaad.2010.07.004 21571173 169. Prakash B , Nadig P , Nayak A Rational prescription for a dermatologist . Indian J Dermatol 2016 ;61 :32 –8 . 10.4103/0019-5154.174017 26955092 170. Nadzam DM Development of medication-use indicators by the Joint Commission on Accreditation of Healthcare Organizations . Am J Hosp Pharm 1991 ;48 :1925 –30 .1928134