
==== Front
J Educ Health Promot
J Educ Health Promot
JEHP
J Edu Health Promot
Journal of Education and Health Promotion
2277-9531
2319-6440
Wolters Kluwer - Medknow India

JEHP-13-126
10.4103/jehp.jehp_308_23
Original Article
Effective factors in planning, implementation, and management of educational program evaluation in medical sciences: A practical guide
Salajegheh Mahla
Department of Medical Education, Medical Education Development Center, Kerman University of Medical Sciences, Kerman, Iran
Address for correspondence: Dr. Mahla Salajegheh, Kerman University of Medical Sciences, Haft-Bagh Highway, Kerman, Iran. E-mail: mahla.salajegheh90@gmail.com
2024
29 4 2024
13 12604 3 2023
22 5 2023
Copyright: © 2024 Journal of Education and Health Promotion
2024
https://creativecommons.org/licenses/by-nc-sa/4.0/ This is an open access journal, and articles are distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License, which allows others to remix, tweak, and build upon the work non-commercially, as long as appropriate credit is given and the new creations are licensed under the identical terms.
BACKGROUND:

Educational program evaluation is a complex issue, and it is essential to have knowledge of the potential challenges and solutions during the whole process. The present study aimed to identify the influential components in planning, implementation, and evaluation management of educational programs in medical sciences and then provide an applied guide to guarantee the best possible evaluation by evaluators of educational programs.

MATERIALS AND METHODS:

This descriptive study was conducted in three steps. First, the effective components in planning, implementation, and evaluation management of educational programs in medical sciences were reviewed. Second, experts’ opinion was asked through a focus group discussion regarding the mentioned components. Third, regarding the complied applied guide, the opinions of 40 medical educationist and program evaluation experts were investigated using a checklist.

RESULTS:

An applied guide for planning, implementation, and evaluation management of educational programs in medical sciences consists of eight stages: determining the evaluation questions and standards, determining the type of information required, determining resources to collect information, determining methods and tools to collect information, determining data analysis methods, determining the timing and frequency of reporting, determining the appropriate ways of reporting, and determining strategies to maintain the cooperation of data sources.

CONCLUSION:

The spread of educational programs in medical sciences universities leads to an increasing need for program evaluation to provide evidence of their effectiveness and improvement. The present research provided an applied guide to make the evaluation of educational programs feasible by using a set of concepts, principles, methods, theories, and models.

Evaluation
guide
medical education
program
OPEN-ACCESSTRUE
==== Body
pmcIntroduction

Evaluating educational programs of medical sciences is one of the most fundamental aspects of educational interventions.[12] Evaluation is an inseparable part of every educational program, and it is a continuous and dynamic method to identify errors in the teaching–learning process.[34] An evaluation is required due to the recent developments in education systems of medical sciences and a huge expenditure and a lot of time on educational programs every year. Therefore, an accurate evaluation is one of the regulators’ concerns that reveal the strengths, weaknesses, and effectiveness of educational programs, as well as finding ways to improve them.[56]

Although a great number of studies have been conducted regarding various approaches to educational programs evaluation, there are few studies presenting solutions for challenges while implementing program evaluation occur[7]. Every challenge presents itself in ways that cannot be expected; therefore, it is difficult for evaluators to have access to detailed guidance[8]. Different approaches and methods to evaluation are mainly generic in nature, and it needs to be clarified what details need to be followed by the evaluators when facing challenges.[9]

Some evaluation challenges introduced by Grandisson et al.[10] in 2014 include scarcity of resources, multiple factors related to the program's effectiveness, and many beneficiaries with unique needs. According to Grandisson et al.,[10] evaluators need to consider various aspects of every program before proceeding to evaluation. Guyadeen et al. (2018)[11] emphasized the importance of providing necessary practical training for the evaluators.

Since no guide has been developed in Iran regarding the ‘implementation’ of educational program evaluation in medical sciences, compiling an applied guide could lead to a major breakthrough. The present study aimed to identify the effective components involved in planning, implementing, and managing the evaluation of educational programs in medical sciences and develop a guide using the effective components to improve program evaluation.

Materials and Methods

Study design and setting

This descriptive study was conducted at Kerman University of Medical Sciences in 2022 in three steps including literature review, focus group, and a survey by a checklist.

Study participants and sampling

For participation in focus group, an e-mail was sent to 10 medical education experts. The participants of the third step were 45 medical education experts and educational program evaluators.

Data collection tool and technique

The first step included a literature review regarding the effective components of designing a guide. Keywords including plan evaluation, planning evaluation, medical evaluation, design management, and program evaluation have been searched through 2010 to 2020 in Medline, Scopus, Web of Sciences, and EMBASE. The presence of keywords in the title and abstract of articles has been considered as an inclusion criterion. The exclusion criteria were unrelated content; studies that did not address the components of planning, implementation, and evaluation management in medical sciences educational programs; studies that had compared different methods of evaluations; and studies that had investigated the effectiveness of different methods of evaluation.

In the second step, one focus group discussion was conducted with several medical education experts. The findings of the previous step were presented and completed during this session by the experts. The experts were informed to attend focus group discussion via e-mail. Therefore, an e-mail was sent to 10 medical education experts. The focus group discussion was held for 2 hours. The collected data regarding the planning, implementation, and evaluation management of medical sciences educational programs were reviewed and discussed. Finally, an eight-stage applied guide was designed and compiled.

In the third step, the compiled guide was given to 45 medical education experts and educational program evaluators as an online checklist with 35 items (23 closed-ended questions and 12 open-ended questions). They were asked to state their opinion regarding the clarity and practicality of each guidance step. Sampling was carried out by census due to the limitation of the experts. The closed-ended questions evaluated the clarity and practicality of the guidance using a dichotomous scale (yes or no), and the open-ended questions were used to collect opinions and suggestions. Data were analyzed using SPSS. Finally, the ultimate guide was developed using the participants’ opinions. The steps of the study are shown in Figure 1.

Figure 1 Steps of the study

Ethical consideration

This study was approved by the Research Ethics Community of Kerman University of Medical Sciences (No. IR.KMU.REC.1400.075.). Participants did not receive any incentives, and participation was voluntary. Informed consent for participation was obtained based on the proposal approved by the ethics committee. The participants were also assured of the confidentiality of their information, and it was explained that the results would only be used for research objectives.

Results

In the first step, 53 articles were found. The titles and the abstract of 49 articles were reviewed. Finally, based on the inclusion criteria, nine articles in Persian and 30 in English were thoroughly studied. Then, the effective components in planning, implementation, and evaluation management of the medical educational programs were established. Table 1 presents the mentioned effective components in planning, implementation, and evaluation management of the medical educational programs divided into three categories.

Table 1 Effective components in planning, implementation, and evaluation management of the medical educational programs

Area	Components	
Evaluation planning	Evaluation questions	
Evaluation standards	
Evaluation implementation	Type of required information	
Appropriate resources for data collection	
Methods and tools for data collection	
Proper condition to collect data	
Data analysis	
Evaluation management	Releasing the results	
Maintaining the cooperation of data sources	

In the second step, the mentioned factors were presented in a focus group discussion for six medical education experts, and based on their opinions, step-by-step practical guidance was developed.

In the third step, 40 checklists were returned out of 45 online checklists (response rate: 88.8%). 45% of the respondents were male, and the rest were female. The work experience mean of the respondents was 8 years, and about 35% of them had less than 5 years of work experience. The minimum work experience was 8 months, and the most extended work experience was 15 years. More than 90% found the content of the guidance clear and unambiguous. 93.6% found the content of the guidance practical. According to the respondents, 74.4% of the least practical belonged to the eighth stage, with the amount of 74.4%. Based on the Chi-square test, there was no significant difference between the frequencies of women's and men's responses regarding the clarity of the overall content of the guidance and its practicality. The frequency distribution of the responses regarding the clarity and practicality of the content revealed that the work experience mean scores of the respondents with the answer ‘yes’ were higher than those with the answer ‘no.’ However, the difference in work experience mean score between the positive and negative answers was not statistically significant.

Finally, the applied guide for planning, implementation, and evaluation management of educational programs in medical sciences includes eight stages: determining the questions and standards, determining the type of information required, determining appropriate resources for data collection, determining tools and methods for data collection, determining the data analysis method, determining the timing and frequency of reporting, determining an appropriate way to present reports, and finally determining strategies to maintain the cooperation of data sources.

Discussion

At the time of implementation or at the end of any educational program such as educational classes, faculty development programs, conferences, and seminars, the people involved including policy-makers, planners, instructors, and evaluation experts evaluate the implemented program. The ultimate end of the evaluation process is to judge and make a decision based on the evidence. In other words, they need to decide whether the program is allowed to be continued or requires modification. The present research aimed to provide an applied guide for professional evaluators to evaluate educational programs carefully. According to the results, the effective components in evaluating educational programs were identified in eight stages of planning, implementation, and evaluation management of medical educational programs.

It is essential to identify questions and standards in relation to evaluation in the area of planning educational programs. When the questions are designated, it will be apparent where the evaluation is headed, and all the following steps will be identified. In regard to the difference and distinction of questions, criteria, and standards of the evaluation, Yarbrough (2017) stated that the evaluation question reflects the purpose of the evaluation, and evaluation criteria state the characteristics of a successful program; last but not least, the evaluation standard states the appropriate characteristics of a program.[12]

According to Nobrega et al. (2021),[13] the process of developing evaluation questions comprises two primary stages, namely, divergent and convergent. In the divergent stage, efforts are made to gather all the questions that seem appropriate to the experts and stakeholders of the program. In the convergent stage, the goal is to categorize and reduce the number of questions based on their importance and relevance to the objective of the evaluation. Finding questions in the divergent stage and selecting the questions in the convergent stage are carried out in collaboration with program stakeholders. The sources of evaluation questions are different. According to the study by Jayaratne in 2016, there are various resources that can be involved in determining evaluation questions. These resources include the questions of stakeholders, program evaluation models, standards, checklists, tools designed for similar evaluations, the perspective and experience of experts, and the evaluator's personal experience and judgment.[14] Once evaluation questions have been selected, it is essential to establish standards for each question. If no standard is defined for a question, it is critical to establish a standard for that question.[12]

It can sometimes be challenging to determine a standard for questions, and it may not be possible to determine a specific level as a standard. Therefore, evaluators need to have a general understanding of standards. When setting standards, evaluators must always be careful to avoid setting standards that are too high or too low.[15] Ahmady et al. (2009)[16] recommended obtaining feedback from stakeholders with different perspectives in order to avoid subjectivity in setting standards.

The implementation steps of the educational program evaluation consist of data collection, analysis, and interpretation. Lemire et al. (2020)[17] stated that there are four essential steps in data collection: determining what information is needed, determining appropriate resources to collect data, determining required methods and tools to collect data, and finally determining appropriate conditions for data collection. According to Nielsen et al. (2022),[18] evaluators must have a plan for coding, organizing, maintaining, retrieving, and analyzing the data. In addition, the interpretation of the findings is one of the important steps in program evaluation since statistical data mean nothing without the right interpretation.

Proper planning and implementation of the evaluation results are helpful; however, if the mentioned results are not effectively reported, the chances of using them cannot be significant. Husereau et al. (2022)[19] conducted a study to consolidated health economic evaluation reporting standards and emphasized various areas including continuous reporting of evaluation results, identifying and applying various reporting methods, identifying audiences, and reporting results based on their requirements and characteristics.

Evaluators must constantly be in contact with the evaluation audience and communicate the results with them. This is an essential matter since it provides the evaluator with an opportunity to have an understanding of their unexpected reaction and a chance to manage it. Moreover, the audiences can have a grasp of the results and a sense of ownership toward it; as a result, they feel motivated to make changes in order to eliminate the imperfection of the program.[20] According to Portell et al. (2015),[21] the timing and frequency of presenting a report depend on the purpose of the evaluation. In formative evaluations, there is more reporting frequency. The timing of intermediate reports can be flexible; it can be either at the end of each stage of the program or at the end of each stage during information collection; it can even be spontaneous and whenever unpredictable results are obtained.

There are various ways to present evaluation reports. Some are less interactive, and some are more interactive. The methods that involve the least interaction between the evaluator and program stakeholders are as follows: reporting through newsletters, summaries, brochures, websites for posting news, or news media.

In the middle of the mentioned spectrum, there are other ways, such as oral presentation, PowerPoint, video report, posters, images, caricatures, animations, and poetry. However, at the end of the spectrum, the most interactive methods involve the most interaction between the evaluator and the stakeholders of the program, including meeting reports, either individually or using simultaneous electronic communication.[2223]

Educational program evaluators must be aware of the audience's needs in proportion to the evaluation reports. Some common mistakes when presenting evaluation reports to various audiences include forgetting a particular audience, not considering their specific needs, and considering too broad or too narrow an audience.[24]

Reporting negative results is of great importance. When presenting adverse reports, it is better to start the report by presenting positive aspects and bringing up negative aspects in face-to-face and friendly meetings; first, an intermediate written report should be provided, and then their reactions should be examined. After this stage, the final report will be sent for review, and then it will be finalized. Moreover, informing stakeholders about negative results cannot be postponed.[25] One way that helps to make negative results more effective while reporting is to ask for the opinions of the audience regarding how to present negative results.[20]

Conclusions

The spread of educational programs in medical sciences universities leads to an increasing need for evaluating the programs to investigate their effectiveness and improvement. Based on a set of concepts, principles, methods, theories, and models in the field of program evaluation, this research provides an applied guide for planning, implementing, and managing educational program evaluation in medical sciences. It consists of eight stages, including setting evaluation questions and standards, identifying required information, selecting appropriate resources for data collection, determining data collection methods and tools, selecting data analysis methods, determining the timing and frequency of evaluation reporting, selecting reporting methods, and identifying strategies to main collaboration among information resources.

Financial support and sponsorship

This study was conducted with the financial support of the Vice Chancellor of Research and Technology, Kerman University of Medical Sciences, under the ethics code of 99001194.

Conflicts of interest

There are no conflicts of interest.

Acknowledgements

The author would like to express her sincere gratitude to all the colleagues who contributed to this study.
==== Refs
References

1. Van Melle E Frank JR Holmboe ES Dagnone D Stockley D Sherbino J , A core components framework for evaluating implementation of competency-based medical education programs Acad Med 2019 94 1002 9 30973365
2. Toosi M Modarres M Amini M Geranmayeh M Context, input, process, and product evaluation model in medical education: A systematic review J Educ Health Promot 2021 10 199 34250133
3. Simonova P Cincera J Kroufek R Krepelkova S Hadjichambis A Active citizens: Evaluation of a community-based education program Sustainability 2019 11 663
4. Sopha S Nanni A The cipp model: Applications in language program evaluation J Asia TEFL 2019 16 1360
5. Gowda D Curran T Khedagi A Mangold M Jiwani F Desai U , Implementing an interprofessional narrative medicine program in academic clinics: Feasibility and program evaluation Perspect Med Educ 2019 8 52 9 30721400
6. Kamali F Yamani N Changiz T Zoubin F Factors influencing the results of faculty evaluation in Isfahan University of Medical Sciences J Educ Health Promot 2018 7 13 29417073
7. Prentice D Engel J Boggs J Does it make a difference? Evaluation of a Canadian poverty reduction initiative Eval Program Plann 2020 80 101817 32172052
8. Smeets RG Hertroijs DF Mukumbang FC Kroese ME Ruwaard D Elissen AM First things first: How to elicit the initial program theory for a realist evaluation of complex integrated care programs Milbank Q 2022 100 89 151
9. Saunders RP Evans MH Joshi P Developing a process-evaluation plan for assessing health promotion program implementation: A how-to guide Health Promot Pract 2005 6 134 47 15855283
10. Grandisson M Thibeault R Hébert M Templeton A Community-based rehabilitation programme evaluations: Lessons learned in the field DCIDJ 2014 25 55 71
11. Guyadeen D Seasons M Evaluation theory and practice: Comparing program evaluation and evaluation in planning J Plan Educ Res 2018 38 98 110
12. Yarbrough DB Developing the program evaluation utility standards: Scholarly foundations and collaborative processes Can J Program Eval 2017 31 284 304
13. Nobrega S Ghaziri ME Giacobbe L Rice S Punnett L Edwards K Feasibility of virtual focus groups in program impact evaluation Int J Qual Methods 2021 20 16094069211019896
14. Jayaratne K Tools for formative evaluation: Gathering the information necessary for program improvement J Ext 2016 54 7
15. Milstein B Wetterhall S Group CEW A framework featuring steps and standards for program evaluation Health Promot Pract 2000 1 221 8
16. Ahmady S Changiz T Brommels M Gaffney FA Thor J Masiello I Contextual adaptation of the personnel evaluation standards for assessing faculty evaluation systems in developing countries: The case of Iran BMC Med Educ 2009 9 1 10 19133146
17. Lemire S Kwako A Nielsen SB Christie CA Donaldson SI Leeuw FL What is this thing called a mechanism? Findings from a review of realist evaluations New Dir Eval 2020 2020 73 86
18. Nielsen SB Lemire S Tangsig S Unpacking context in realist evaluations: Findings from a comprehensive review Evaluation 2022 28 91 112
19. Husereau D Drummond M Augustovski F de Bekker-Grob E Briggs AH Carswell C , Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 explanation and elaboration: A report of the ISPOR CHEERS II good practices task force Value Health 2022 25 10 31 35031088
20. Moscoso SC Chaves SS Vidal MP Argilaga MTA Reporting a program evaluation: Needs, program plan, intervention, and decisions Int J Clin Health Psychol 2013 13 58 66
21. Portell Vidal M Anguera Argilaga MT Chacón Moscoso S Sanduvete Chaves S Guidelines for reporting evaluations based on observational methodology Psicothema 2015 27 283 9 26260937
22. Smith KG Alexander K Cleland J Opening up the black box of a gateway to medicine programme: A realist evaluation BMJ Open 2021 11 e049993
23. Foo J Cook DA Walsh K Golub R Abdalla ME Ilic D , Cost evaluations in health professions education: A systematic review of methods and reporting quality Med Educ 2019 53 1196 208 31402515
24. Norris JM Plonsky L Ross SJ Schoonen R Guidelines for reporting quantitative methods and results in primary research Language Learning 2015 65 470 6
25. Hyland K Activity and evaluation: Reporting Practices in Academic Writing Academic Discourse Routledge 2014 125 40
