==== Front Healthc Inform ResHealthc Inform ResHIRHealthcare Informatics Research2093-36812093-369XKorean Society of Medical Informatics 10.4258/hir.2018.24.3.157Original ArticleSafety and Usability Guidelines of Clinical Information Systems Integrating Clinical Workflow: A Systematic Review Lee Yura MDPhD1Jung Min-Young RNMS1Shin Gee Won MS2Bahn Sangwoo PhD3Park Taezoon PhD4Cho Insook RNPhD5Lee Jae-Ho MDPhD161 Department of Biomedical Informatics, Asan Medical Center, Seoul, Korea.2 Department of Industrial Engineering, Human Interface System Lab, Seoul National University, Seoul, Korea.3 Department of Industrial and Management Systems Engineering, Kyung Hee University, Seoul, Korea.4 Department of Industrial and Information Systems Engineering, Soongsil University, Seoul, Korea.5 Department of Nursing, Inha University, Incheon, Korea.6 Department of Emergency Medicine, University of Ulsan Collage of Medicine, Seoul, Korea.Corresponding Author: Jae-Ho Lee, MD, PhD. Department of Emergency Medicine, University of Ulsan College of Medicine, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, Korea. Tel: +82-2-3010-3350, jaeholee@amc.seoul.kr7 2018 31 7 2018 24 3 157 169 03 5 2018 08 7 2018 12 7 2018 © 2018 The Korean Society of Medical Informatics2018The Korean Society of Medical InformaticsThis is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.Objectives The usability of clinical information systems (CISs) is known to be an essential consideration in ensuring patient safety as well as integrating clinical flow. This study aimed to determine how usability and safety guidelines of CIS consider clinical workflow through a systematic review in terms of the target systems, methodology, and guideline components of relevant articles. Methods A literature search was conducted for articles published from 2000 to 2015 in PubMed, Cochrane, EMBASE, Web of Science, and CINAHL. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement method was employed. Articles containing recommendations, principles, and evaluation items for CIS usability and safety were included. The selected articles were classified according to article type, methodology, and target systems. Taking clinical workflow into consideration, the components of guidelines were extracted and classified. Results A total of 7,401 articles were identified by keyword search. From the 76 articles remaining after abstract screening, 15 were selected through full-text review. Literature review (n = 7) was the most common methodology, followed by expert opinions (n = 6). Computerized physician order entry (n = 6) was the most frequent system. Four articles considered the entire process of clinical tasks, and two articles considered the principles of the entire process of user interface affecting clinical workflow. Only two articles performed heuristic evaluations of CISs. Conclusions The usability and safety guidelines of CISs need improvement in guideline development methodology and with consideration of clinical workflow. Hospital Information SystemsPatient SafetyUser-Computer InterfaceGuidelineWorkflowMinistry of Health and Welfarehttp://dx.doi.org/10.13039/501100003625HI15C1101 ==== Body I. Introduction Many errors regarding clinical information systems (CISs) have been reported [1234]. Research on errors and potential hazards has led to review and updating of CISs to promote patient safety [567]. Efforts to reduce errors have assessed CISs in light of human factors [8]. As a result, guidelines have been developed and distributed to consider usability from the CIS design stage. However, efforts to pursue patient safety have sometimes had conflicting results regarding usability. An alert function that can filter out simple input errors may interfere with the flow of work, which may adversely affect usability [91011]. On the other hand, functions to improve usability can cause errors. For a novice user, shortcuts or abbreviations for work efficiency could threaten patient safety. Therefore, user-centered CIS design guidelines are needed, not only to prevent errors but also to improve clinical workflow [12131415]. The accumulation of structured (coded) data enables the development of decision support rules. Improvements in network speed allow immediate feedback to users, with processing of evidence-based medicine and patient data. Consequently, system-level controls to reduce user mistakes (or for best practices) have become easier. Moreover, professional considerations of usability are needed as CISs become more complex and sophisticated because novice personnel regularly enter the field and must be adapted to the system [161718]. We should consider the diverse functions of CISs and various fields of labor in CIS design and maintenance, as well as upgrades in CIS guidelines. We conducted this study to help establish guidelines to improve the safety and usability of CIS through a systematic review and to analyze how the clinical workflow and usability principles are reflected differently in existing articles [121920]. We divided the steps of clinical workflow and performed a guideline review. Furthermore, this review sought clues to balancing CIS usability and safety. II. Methods This review of CIS guidelines was based on a systematic review of the literature based on the results of searches on the related keywords. Detailed recommendations and principles of evaluation were extracted by reviewing the selected articles, whose components were classified and reassembled according to various criteria referenced in the selected literature to analyze how they affect clinical workflow. The present study protocol was reviewed and approved by the Institutional Review Board of Asan Medical Center (No. 2016-0980). 1. Systematic Review of Relevant Articles Literature searches were performed in PubMed, Cochrane, EMBASE, Web of Science, and CINAHL using keywords related to health information technology (HIT), safety, and usability in guideline development published in English between January 2000 and December 2015 (Table 1). The protocol for this review was guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. The keywords used in the search were selected through discussions by the research team. Keywords were classified into three categories: guideline, HIT, and safety/usability. Each category was searched using OR for keywords and synonyms and AND between categories. Articles containing recommendations, principles, and evaluation items for CIS design, usage, or evaluation were included. On the other hand, those with ‘content vague or with too narrow focus’, ‘issues irrelevant to safety or usability principle’, ‘no suggestion for guideline/recommendation’, or ‘unable to find full-text article’ were excluded. Two researchers independently checked the titles and abstracts of the searched articles and conducted the first selection (Figure 1). Upon agreement between these two researchers, the selection of the documents to be included in the final evaluation was made through discussion with a third researcher. After securing the full text for the final selection, secondary review was performed independently by the first two researchers. Conflicts were resolved, and the final selection for the full literature review was made through a discussion by all three researchers. In case of non-agreement, the research team reached a decision through consultation. 2. Review of Selected Articles and Extraction of the Guideline Components The selected articles were reviewed in terms of objectives, the type of articles, and nationality. The target system and methodology of the articles were also classified according to whether they presented a guideline or principle based on existing knowledge (e.g., literature review), empirical content (e.g., expert opinion or group discussion), user opinion (e.g., user survey or testing), or heuristic evaluations. The guidelines, recommendations, principles, or evaluation items from the articles were then extracted as guideline components. 3. Review and Re-classification of Guideline Components Affecting Clinical Workflow We stratified the clinical workflow according to the following types of actions. A user identifies and selects patients’ data, records medical documents, orders medications or investigations, gets feedback from the system by alert or Clinical Decision Support (CDS), and manages and maintains the system [1221222324]. Therefore, the guideline components were classified according to clinical tasks (data identification & selection, document entry, order entry, clinical decision support & alert, and management) and usability principles (screen recognition, data view & entry, running & control, and feedback) following considerations of clinical workflow. The research team determined the final categories and methods for classifying the guideline components through discussion. III. Results Among a total of 7,401 searched articles, 15 articles were finally selected through the systematic review [7910172526272829303132333435]. Table 2 summarizes the characteristics of the studies included in this systematic review. All but one of the studies were written in the United States (73%, 11/15) or Canada (20%, 3/15); the other was from the UK (Table 2). Almost half (47%, 7/15) of the articles were original research papers, followed by perspective, opinion, and commentary (5/15) and review articles (3/15). The article types were assigned based on information from the journals. For the target system, Computerized Physician Order Entry (CPOE) was the most frequent system, whereas Electronic Medical Record (EMR) was the least frequent (Table 3). Article review was the most common methodology (7/15), followed by expert opinion (6/15). An article review was followed by additional analysis in the majority of cases (4/7). Three articles concerned the results of user testing or surveys, and only two articles conducted heuristic evaluations. Clinical components such as ‘drug-patient age checking’ were classified under ‘clinical tasks’, and usability components such as ‘clearly legible font’ were classified under ‘usability principles’ [1034]. There were some conflicts within a given category. For example, default values for medication prescription are sometimes recommended (“The system should provide for selection from the dosages and forms that are available and appropriate for a given medication,” by Bell et al. [26]) and sometimes not (“System should limit or not use defaults for medications,” by Carvalho et al. [28]). A brief explanation, such as ‘Only enter default values for drug, dose, frequency, and route if it will always be correct’, balanced safety and usability [7]. Table 4 shows the results of the categorization of guideline components according to clinical workflow. Four of the 15 selected articles [10172632] were classified as applicable to the entire process of the clinical tasks affecting clinical workflow. For example, “Content should be limited to 1–2 lines, with a justification separated by white space” by Horsky et al. [10] for data identification & selection; “Monitor use of C/P functions in record preparation and limit use of ‘boilerplate’ content across records” by Zahabi et al. [17] for document entry; and “Conducts a verification process to ensure that all medications comply with recommended dosing based on current evidence-based literature” by the Institute for Safe Medication Practices (ISMP) [32] for order entry and CDS & alert. Within the clinical tasks, CDS and alert (13/15) and system management (9/12) were commented on by a majority of the articles. General principles of alert systems, such as alert priority or reduction of alert fatigue, were most frequently discussed in the articles (Table 5). Two articles [1732] were classified as applicable to the entire process of selecting the usability principles affecting clinical workflow (Table 5). Running & control was the most frequently considered principle (10/15), and data view & entry was the least (4/15). The article by Zahabi et al. [17] concerned the clinical tasks and usability principles affecting clinical workflow. The methodologies of the 5 commented articles were article review [1017], group discussion [2632], and expert opinion [31]. IV. Discussion In this study, we selected articles presenting guidelines or principles for CIS safety and usability that should be taken into account in the design, use, and management of such systems. We reviewed the target system and methodology and reviewed and summarized the guidelines and evaluation items to date. Also, we divided the steps of clinical workflow into the aspects of clinical tasks and usability principles. Articles focused on specific tasks (CDS and alert) (13/15). Also, among the usability principles, the components of data view & entry were relatively small (4/15). Articles showed a lack of diversity in clinical task and principle for each step of clinical workflow. All articles with inclusive clinical workflow components were based on existing knowledge or experience [10172631]. To our knowledge, no study has reviewed the CIS safety and usability guidelines according to the detailed steps of clinical workflow. 1. Balance in Conflicts of Safety and Usability of CIS Although improvement of usability is directed toward pursuing patient safety, some cases had conflicts, as mentioned above regarding default values for medication prescription. Recommendations like “Omit items for which the information is not available to the user,” which are intended to reduce cognitive load, could threaten patient safety if clinical workflow is not taken into consideration. Similarly, “Cluster related information on the same screen” [17] and “Avoid too much information on the screen at one time” [29] seem to be mutually contradictory without a consideration of the clinical situation. Participation of clinicians and feedback to users can mitigate the conflicts in safety versus usability, as comments on the role of clinicians in system adaptation and order set pointed out [3235]. Some override functions have the risk of bypassing critical alerts, though they prevent alert fatigue, which reduces workflow efficiency [11121314262728]. However, profiling and reporting the history of alert overrides can be helpful in cases where critical alerts are of overridden [2627]. Therefore, the consideration of clinical workflow from the user's point of view is important to balance safety and usability. 2. Challenges of Previous Studies: Localized Methodologies and Target Systems As the terms of the systematic review were set to English articles, and all the selected articles were North American and UK studies. Among the selected articles were seven original research articles according to the publishers' classifications, but expert survey/group discussions constituted the majority of these articles. Few methodologies of heuristic evaluation were given [2], and only one article conducted a retrospective cohort study [35]. The majority included existing knowledge (article review, expert opinion, or group discussion), and there were few articles on user testing/surveys. Much of the content concerned CDS, actions, and management, but that on data presentation was relatively small. Only one-third (5/15) of the articles included all the steps of clinical tasks and/or usability principles affecting clinical workflow. Moreover, these 5 articles were based on existing knowledge, not up-to-date user experiences or experimental evidence. Article reviews and empirical knowledge are essential for the improvement of past error and usability problems. However, considering improvements in the speed of system performance and the development of infrastructure such as a CIS network, more active experiments and studies of systems-in-use and/or systems-in-advance will be needed. There was no specific recommendation or consideration of the size or specificity of the medical institutions except for one document related to a GP system in primary care, Avery et al. [27] (Table 2). The use of computers in medical institutions of various sizes has expanded the management of patient data. The need for the development of principles of information systems to ensure the efficient and safe exchange of information with primary care institutions and hospitals has increased. Our findings were similar to those of previous systematic reviews on CIS safety/usability, in that most of the articles lacked a consideration of overall clinical workflow. In methodology, the majority were based on existing knowledge or empirical content and lacked an explicit theoretical framework or model, as Yen and Bakken [19] commented. 3. Limitations In this study, the components of more specific functions or tasks (e.g., patient lists, dashboards, or override alerts) were not included. Also, non-English articles and guidelines were not included, and as this study was a part of the guideline development process, experimental articles not containing guidelines or principles were excluded. This factor might have served to exclude more experimental research articles. 4. Conclusion There was a lack of consideration of the entire clinical workflow in the selected articles. Also, in many cases, guidelines were developed through the synthesis of existing knowledge rather than through user testing or heuristic evaluations. Development of CIS guidelines affecting clinical workflow is needed for usability and patient safety. To promote the safety and usability of CIS, more user-oriented guidelines that take into account the clinical work-flow are needed. Acknowledgments This study was supported by a grant of the Korean Health Technology R&D Project, Ministry of Health & Welfare, Korea (No. HI15C1101). Conflict of Interest: No potential conflict of interest relevant to this article was reported. Figure 1 Flowchart of the systematic review. Table 1 Selected keywords and their categories for the literature search Each category was searched by OR for keywords and synonyms, and by AND between categories. Table 2 Summary of the selected articles CDS: Clinical Decision Support, CPOE: Computerized Physician Order Entry, EMR: Electronic Medical Record, EHR: Electronic Health Record, ISMP: Institute for Safe Medication Practices, VA CPRS: US Department of Veterans Affairs Computerized Patient Record System. Table 3 Target systems and methodologies of selected articles CPOE: Computerized Physician Order Entry, CDS: Clinical Decision Support, EMR: Electronic Medical Record, CIS: Clinical Information Systems, ISMP: Institute for Safe Medication Practices. Table 4 Components of guideline/principles for clinical tasks or usability principles affecting clinical workflow CDS: Clinical Decision Support, ISMP: Institute for Safe Medication Practices. Table 5 Guideline components for clinical tasks and usability principles considering workflow in selected articles EMR: Electronic Medical Record, CDS: Clinical Decision Support. ==== Refs 1 Menachemi N Collum TH Benefits and drawbacks of electronic health record systems Risk Manag Healthc Policy 2011 4 47 55 22312227 2 Friedberg MW Chen PG Van Busum KR Aunon F Pham C Caloyeras J Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy Rand Health Q 2014 3 4 1 28083306 3 Campbell EM Sittig DF Ash JS Guappone KP Dykstra RH Types of unintended consequences related to computerized provider order entry J Am Med Inform Assoc 2006 13 5 547 556 16799128 4 Harrison MI Koppel R Bar-Lev S Unintended consequences of information technologies in health care: an interactive sociotechnical analysis J Am Med Inform Assoc 2007 14 5 542 549 17600093 5 Blumenthal D Tavenner M The “meaningful use” regulation for electronic health records N Engl J Med 2010 363 6 501 504 20647183 6 Sengstack P CPOE configuration to reduce medication errors J Healthc Inf Manag 2010 24 4 26 34 7 Hyman D Laire M Redmond D Kaplan DW The use of patient pictures and verification screens to reduce computerized provider order entry errors Pediatrics 2012 130 1 e211 e219 22665415 8 Brown CL Mulcaster HL Triffitt KL Sittig DF Ash JS Reygate K A systematic review of the types and causes of prescribing errors generated from using computerized provider order entry systems in primary and secondary care J Am Med Inform Assoc 2017 24 2 432 440 27582471 9 Bates DW Kuperman GJ Wang S Gandhi T Kittler A Volk L Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality J Am Med Inform Assoc 2003 10 6 523 530 12925543 10 Horsky J Schiff GD Johnston D Mercincavage L Bell D Middleton B Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions J Biomed Inform 2012 45 6 1202 1216 22995208 11 Prgomet M Li L Niazkhani Z Georgiou A Westbrook JI Impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay, and mortality in intensive care units: a systematic review and meta-analysis J Am Med Inform Assoc 2017 24 2 413 422 28395016 12 Eisenberg F Barbell AS Computerized physician order entry: eight steps to optimize physician workflow J Healthc Inf Manag 2002 16 1 16 18 13 Tu SW Musen MA Shankar R Campbell J Hrabak K McClay J Modeling guidelines for integration into clinical workflow Stud Health Technol Inform 2004 107 Pt 1 174 178 15360798 14 Niazkhani Z Pirnejad H Berg M Aarts J The impact of computerized provider order entry systems on inpatient clinical workflow: a literature review J Am Med Inform Assoc 2009 16 4 539 549 19390113 15 Khajouei R Jaspers MW The impact of CPOE medication systems' design aspects on usability, workflow and medication orders: a systematic review Methods Inf Med 2010 49 1 3 19 19582333 16 Ellsworth MA Dziadzko M O'Horo JC Farrell AM Zhang J Herasevich V An appraisal of published usability evaluations of electronic health records via systematic review J Am Med Inform Assoc 2017 24 1 218 226 27107451 17 Zahabi M Kaber DB Swangnetr M Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation Hum Factors 2015 57 5 805 834 25850118 18 Middleton B Bloomrosen M Dente MA Hashmat B Koppel R Overhage JM Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA J Am Med Inform Assoc 2013 20 e1 e2 e8 23355463 19 Yen PY Bakken S Review of health information technology usability study methodologies J Am Med Inform Assoc 2012 19 3 413 422 21828224 20 Zhang J Johnson TR Patel VL Paige DL Kubose T Using usability heuristics to evaluate patient safety of medical devices J Biomed Inform 2003 36 1-2 23 30 14552844 21 Lauesen S Younessi H Six styles for usability requirements Proceedings of the 4th International Workshop on Requirements Engineering: Foundation for Software Quality (REFSQ) 1998 Jun 8-9 Pisa, Italy 155 166 22 Dix A Human-computer interaction Liu L Ozsu MT Encyclopedia of database systems Boston (MA) Springer 2009 1327 1331 23 Nielsen J Heuristic evaluation Nielsen J Marck RL Usability inspection methods New York (NY) John Wiley & Sons 1994 25 62 24 Nielsen J Usability engineering San Francisco (CA) Morgan Kaufmann Publishers 1993 25 Nolan TW System changes to improve patient safety BMJ 2000 320 7237 771 773 10720364 26 Bell DS Marken RS Meili RC Wang CJ Rosen M Brook RH Recommendations for comparing electronic prescribing systems: results of an expert consensus process Health Aff (Millwood) 2004 Suppl Web Exclusives W4-305-17 27 Avery AJ Savelyich BS Sheikh A Cantrill J Morris CJ Fernando B Identifying and establishing consensus on the most important safety features of GP computer systems: e-Delphi study Inform Prim Care 2005 13 1 3 12 15949170 28 Carvalho CJ Borycki EM Kushniruk A Ensuring the safety of health information systems: using heuristics for patient safety Healthc Q 2009 12 Spec No Patient 49 54 19667777 29 Corrao NJ Robinson AG Swiernik MA Naeim A Importance of testing for usability when selecting and implementing an electronic health or medical record system J Oncol Pract 2010 6 3 120 124 20808553 30 Chan J Shojania KG Easty AC Etchells EE Usability evaluation of order sets in a computerised provider order entry system BMJ Qual Saf 2011 20 11 932 940 31 Zopf-Herling KM Enhancing usability in EMR screen design Comput Inform Nurs 2011 29 12 679 691 22173377 32 Institute for Safe Medication Practices Guidelines for standard order sets [Internet] Horsham (PA) Institute for Safe Medication Practices 2010 cited at 2018 May 4 Available from https://www.ismp.org/guidelines/standard-order-sets 33 McGreevey JD 3rd Order sets in electronic health records: principles of good practice Chest 2013 143 1 228 235 23276846 34 Vartian CV Singh H Russo E Sittig DF Development and field testing of a self-assessment guide for computer-based provider order entry J Healthc Manag 2014 59 5 338 352 25647953 35 Green LA Nease D Jr Klinkman MS Clinical reminders designed and implemented using cognitive and organizational science principles decrease reminder fatigue J Am Board Fam Med 2015 28 3 351 359 25957368