
==== Front
Can Med Educ J
Can Med Educ J
CMEJ
Canadian Medical Education Journal
1923-1202
Canadian Medical Education Journal

CMEJ-15-050
10.36834/cmej.77188
Brief Reports
Is Competency-Based Medical Education being implemented as intended? Early lessons learned from Physical Medicine and Rehabilitation
L’approche par compétences en formation médicale est-elle mise en œuvre tel que prévu? Premiers enseignements tirés de la médecine physique et réadaptationTrier Jessica 1
Askari Sussan 1
Hanmore Tessa 123
Thompson Heather-Ann 1
McGuire Natalie 45
Braund Heather 45
Hall Andrew Koch 67
McEwen Laura 89
Dalgarno Nancy 4
Dagnone Jeffrey Damon 610
1 Department of Physical Medicine and Rehabilitation, Queen’s University, Ontario, Canada;
2 Department of Ophthalmology, Queen's University, Ontario, Canada;
3 Department of Psychiatry, Queen’s University, Ontario, Canada;
4 Office of Professional Development and Educational Scholarship, Queen’s University;
5 Department of Biomedical and Molecular Sciences, Queen's University, Ontario, Canada;
6 Royal College of Physicians and Surgeons of Canada, Ontario, Canada;
7 Department of Emergency Medicine, University of Ottawa, Ontario, Canada;
8 Department of Pediatrics, Queen’s University, Ontario, Canada;
9 Postgraduate Medical Education, Queen's University, Ontario, Canada;
10 Department of Emergency Medicine, Queen’s University, Ontario, Canada
Correspondence to: Jessica Trier; 752 King St. West, Kingston, ON, Canada, K7M 4X3; phone (613) 544-4900 ext. 53286; fax (613) 544-4614; email: jessica.trier@queensu.ca
13 5 2024
30 8 2024
8 2024
15 4 5055
© 2024 Trier, Askari, Hanmore, Thompson, McGuire, Braund, Koch Hall, McEwen, Dalgarno, Dagnone; licensee Synergies Partners.
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an Open Journal Systems article distributed under the terms of the Creative Commons Attribution License. (https://creativecommons.org/licenses/by-nc-nd/4.0) which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is cited.
Background

As competency-based medical education (CBME) curricula are introduced in residency programs across Canada, systematic evaluation efforts are needed to ensure fidelity of implementation. This study evaluated early outcomes of CBME implementation in one Canadian Physical Medicine and Rehabilitation program that was an early adopter of CBME, with an aim to inform continuous quality improvement initiatives and CBME implementation nationwide.

Methods

Using Rapid Evaluation methodology, informed by the CBME Core Components Framework, the intended outcomes of CBME were compared to actual outcomes.

Results

Results suggested that a culture of feedback and coaching already existed in this program prior to CBME implementation, yet faculty felt that CBME added a framework to support feedback. The small program size was valuable in fostering strong relationships and individualized learning. However, participants expressed concerns about CBME fostering a reductionist approach to the development of competence. Challenges existed with direct observation, clear expectations for off-service training experiences, and tracking trainee progress. There was trepidation surrounding national curricular change, yet the institution-wide approach to CBME implementation created shared experiences and a community of practice.

Conclusions

Program evaluation can help understand gaps between planned versus enacted implementation of CBME, and foster adaptations to improve the fidelity of implementation.

Contexte

À mesure que les programmes d’approche par compétences (APC) en formation médicale sont introduits dans les programmes de résidence au Canada, des efforts d'évaluation systématiques sont nécessaires pour assurer la fidélité de la mise en œuvre. Cette étude a évalué les premiers résultats de la mise en œuvre de l’APC en formation médicale dans un programme canadien de médecine physique et réadaptation, qui a été un des premiers à adopter l’APC, dans le but d'orienter les initiatives d'amélioration continue de la qualité et de la mise en œuvre de l’APC à l'échelle nationale.

Méthodes

En utilisant une méthodologie d’évaluation rapide, fondée sur le cadre des composantes de base de l’APC en formation médicale, les résultats escomptés de l’APC ont été comparés aux résultats réels.

Résultats

Les résultats suggèrent qu'une culture de la rétroaction et de l’encadrement existait déjà dans ce programme avant la mise en œuvre de l’APC, mais le corps professoral a estimé que l’APC en formation médicale a ajouté un cadre pour soutenir cette rétroaction. La petite taille du programme a permis de favoriser des relations solides et un apprentissage individualisé. Cependant, les participants ont exprimé des inquiétudes quant au fait que l’APC favorise une approche réductionniste du développement des compétences. L'observation directe, les attentes claires en matière d'expériences de formation hors de l’environnement clinique et le suivi des progrès des résidents posent problème. Le changement de programme national a suscité des inquiétudes, mais l'approche institutionnelle de la mise en œuvre de l’APC a permis de partager des expériences et de créer une communauté de pratique.

Conclusions

L'évaluation des programmes peut aider à comprendre les écarts entre la mise en œuvre planifiée et effective de l’APC en formation médicale, et de favoriser les adaptations pour améliorer le respect des conditions de mise en œuvre.

Funding: This project was funded by the PGME Special Bursary Grant.
==== Body
pmcIntroduction

The evaluation of competency-based medical education (CBME) implementation is necessary to articulate the fidelity and challenges of implementation and inform future change.1–3 For example, previous evaluations of CBME implementation prompted adaptations to improve feedback to trainees.4,5 The impact of CBME’s volume of assessments on trainee wellness has also raised concerns about a reductionist approach to competence.6,7 Thus, effective adoption of CBME depends on systematic efforts to evaluate the strengths and challenges of early implementation with an aim towards improvement.8

Evaluating the fidelity of CBME implementation, or the extent to which the enacted and intended curricula align,4 also reveals how the core components of CBME are actualized.9 Assessing fidelity allows program-level adaptations and guidance for programs transitioning to CBME. Evaluation is also critical as accreditation requirements now include continuous quality improvement (CQI) metrics.3 While the body of literature is growing regarding the transition to CBME and potential benefits of program evaluation, we still do not fully understand how programs can best adapt through iterative change.

The Physical Medicine and Rehabilitation (PM&R) program at Queen’s University, in Kingston, Canada, transitioned to CBME in 2017 as part of an institution-wide initiative.10 As all Canadian PM&R programs transitioned to the Royal College’s (RC) version of CBME, Competence by Design (CBD), in July 2020, the Queen’s PM&R program was in a unique position to report on early findings and share lessons learned to programs who transitioned later. The objective of this program evaluation was to understand CBME implementation outcomes in PM&R at Queen’s University, to compare actual outcomes to intended outcomes, and to inform CQI initiatives locally and nationally.

Context

The PM&R residency program at Queen’s University is a five-year program with eight faculty members, one program administrator (PA), and six trainees. Implementation of CBME in 2017 followed the key principles and design of the RC’s CBD model2 at the institutional level rather than the national level, until the national transition to CBD in 2020. The CanMEDS Framework11 articulates stage-specific competencies and milestones across four stages of training. Milestones are individual abilities along a developmental continuum, whereas entrustable professional activities (EPAs) are the professional tasks of a physician.12 Assessment forms on the online learning management system document EPA observations, and there are various ways that these can be completed (eg., sending a blank form to faculty, completing a form in the moment, or residents pre-populating the form then sending to faculty). Trainees receive coaching in the moment from faculty, and coaching over time from an Academic Advisor (AA).13 Educational oversight and support is provided by the Program Director (PD), CBME Lead, PA, and Educational Consultant (EC).14 The competence committee (CC) makes decisions about trainee progress and promotion.

Methods

This study was part of an institution-wide evaluation of CBME implementation at Queen’s University. Following institutional ethics approval (File #6015151), all PM&R program stakeholders were invited to participate, and written informed consent was obtained.

Rapid Evaluation (RE) methodology articulates a stepwise approach to CBME program evaluation with an aim to foster and document progression towards deep system change.4,15 The steps include describing ideal implementation, measuring enacted implementation, and comparing the two to inform adaptations.4 The evaluation aimed to measure program-level fidelity of CBME implementation15,16 and early outcomes.

The evaluation team was comprised of institutional CBME leaders, PM&R program team members (PD, CBME lead, EC, and PA), as well as three education researchers with no program affiliations. The program context and intended outcomes were described by the program team. All program stakeholders were invited via email to participate in the study. Focus groups were separated by role (trainees and faculty), while program leaders and staff were invited to one-on-one interviews to mitigate influence within focus groups. Focus groups were conducted in person, by two of the education researchers (NM and HB). Interviews were conducted in person or over the phone with one of the education researchers (NM). Following verbal consent, all focus groups and interviews were audio-recorded and followed a semi-structured interview guide.

Analyses were abductive,16,17 where the study team used conventional content analysis18 to identify themes, then checked the themes against existing data and theories such as the CBME Core Components Framework (CCF)9, and then went back to the data to create a summary of planned versus enacted implementation. Analyses were completed by the two education researchers involved in data collection (NM & HB). The researchers independently analyzed one interview and one focus group using line-by-line coding and met to discuss their findings. From this discussion, a consensus-built codebook was generated and utilized by NM to code the remaining transcripts. Results from each interview and focus group were then triangulated and reviewed with the evaluation team and PM&R program leaders. Key themes were identified and used to compare planned versus enacted CBME implementation and outcomes. All results were then reported back to program stakeholders during an education rounds session for member-checking, additional comments, and discussion on planned adaptations.19

Results

Fourteen of seventeen (82%) program stakeholders participated in this study (Table 1). One focus group was conducted with CBME trainees (CT; n = 3/3, 100%) and traditional (non-CBME) trainees (NCT; n = 3/3, 100%). A separate focus group comprised faculty (F; n = 3) while two faculty program leaders including PD and CBME lead (PL; n = 3) participated through individual interviews, for a total of 6/8 (75%) of faculty represented. Program staff included PA and EC (PL; n = 2/2, 100%) participated through individual interviews.

Table 1 Study participants by stakeholder group

Stakeholder Group	Proportion of population represented	
Non-CBME trainees (NCT)	n = 3/3 (100%)	
CBME trainees (CT)	n = 3/3 (100%)	
Faculty members (F)a	n = 6/8 (75%)	
Program leaders (PL)a
Program staff
CC members a	n = 3/3 (100%)
n = 2/2 (100%)
n = 5/6 (83%)	
Abbreviations: CBME, competency-based medical education; CC, competence committee. aNote a total of six faculty participated; program leaders and CC members shown in the table held multiple roles and are represented in the faculty count.

Overall, participants were supportive of the transition to CBME, acknowledging the support from the CBME lead, EC, department, postgraduate medical education, and faculty development initiatives. Participants believed in the theory underpinning CBME but found implementation challenging, particularly the time commitment.

Results suggested that a culture of feedback and coaching existed in this program before CBME implementation: “A lot of the feedback I would have [received] regardless … this department is very committed to teaching and coaching” (CT1). Although a culture of feedback existed, faculty felt that EPAs provided a framework to support feedback: “I try to talk to [trainees] at the beginning of their rotation about what their goals are … Now it is also about what EPAs are you working on ... It has provided a bit more structure” (F4). While EPAs facilitated more structured feedback, completing, and tracking assessments remained challenging.

Trainees pre-populating assessment forms was a matter of debate. Trainees felt that if they pre-populated assessment forms, they might not get additional feedback: “If I ever did the option where you could fill it out yourself, … 99% of the time [the faculty] would just submit whatever I wrote” (CT4). However, faculty found pre-populated assessment forms helpful to facilitate documentation and incorporate the trainee’s self-assessment.

While CBME clarified expectations, trainees raised concerns that assessment requirements promoted a reductionist, checklist-oriented approach to the development of competence. As one resident stated, “It becomes this checklist thing. …. [if something] is not on the checklist it is going to be a bit harder to make a case [to capture] those [skills] that might be really important” (NCT6). Trainees felt expectations during off-service experiences were unclear: “You go through these off-service rotations, and you don’t have a lot of things to get specific feedback on … it made me a bit anxious because I am like, am I actually doing anything?” (CT1). Trainees also expressed mixed feelings about EPAs specific to off-service rotations. Others expressed that not having designated off-service EPAs was appreciated as there was little control over clinical experiences.

Direct observation was challenging due to lengthy encounters: “We spend a lot of time with each patient … Am I going to sit for 45 minutes and watch the resident take a history?” (PL1). Additional challenges included time to complete assessments, difficulty interpreting stage-specific entrustment anchors, difficulty understanding which EPAs were relevant in which clinical context, and problems tracking trainee progress.

The CC process provided intentional review of all trainees, and more documentation of performance: “The nature of the comments I think are very similar ... I just don’t think we documented it in such great detail” (F3). While CC members felt the CC process was strong, trainees were relatively unaware of the process.

The small program size nurtured strong relationships and individualized learning but posed challenges to faculty workload. Participants questioned the sustainability of CBME given the workload resulting from review and synthesis of assessment data for CC meetings and the resources required for another transition to CBD. However, the institutional approach to implementation fostered shared experiences and a community of practice.20

Participants raised concerns regarding the nationwide shift to CBD. While they expressed relative comfort with the new curriculum, there was trepidation about a second transition. Lack of sustained funding was a concern, particularly in light of the upcoming transition to CBD and the previous influx of funding for CBME implementation.

Current adaptations and future planning

Table 2 utilizes the CCF framework to compare the expected outcomes of the program with the identified outcomes and includes program-level adaptations to address gaps.

Table 2 Core components, with description of expected outcomes, identified implementation outcomes, and adaptations made by the program

Core Component	Expected Outcomes	Identified Outcomes	Adaptations	
Outcomes Competencies	Trainees and faculty understand the basic rationale for CBME and entrustment scores; program offers ongoing development opportunities.	CBME is good in theory but challenging and time consuming in practice.
Support from the program CBME lead, department, PGME, and EC was critical to generating buy in and successful implementation.
Several faculty development initiatives were implemented prior to the evaluation. Concerns remained about the shift to CBD, funding, and the potential loss of the EC role.	Ongoing education about the relevance and changes related to CBME (and CBME transition to CBD) during education rounds, journal clubs, and email updates.
Transition and sustainability plan for current EC role.	
Sequenced Progression	Trainees and faculty understand and use milestones and EPAs for the appropriate stage of training.	EPAs are straightforward; however, milestones are not often used and are not highly valued.
There are challenges interpreting and using stage-specific entrustment anchors.	Entrustment scales have been adjusted where appropriate (i.e. will use O-SCORE for procedure forms).
Continued faculty development about stage-specific entrustment.	
Tailored Learning Experiences	RTEs are mapped appropriately; faculty are aware of RTEs & individualize learning; trainees seek learning opportunities related to EPAs.	EPAs and expectations are straightforward; however, there is little direction for what should be accomplished during off-service experiences.	One-page summary to clarify expectations of off-service rotations.	
Competency-Focused Instruction	Faculty directly observe trainees and give constructive feedback; trainees track data on LMS to identify learning needs.	The benefit of a small program is a lot of 1:1 interaction, and the development of strong relationships.
Due to the length of patient encounters, it is challenging to provide direct observation.
Challenges with the LMS: trainees find it difficult to understand which EPAs they needed to complete; faculty find it difficult to use the LMS to complete assessments; program leaders find it time consuming to review assessments and synthesize data for CC meetings.	Faculty have been prompted to allocate clinic time in a way that will allow for direct observation. Reminders will also be given periodically.
EPA tracking sheets have been created.	
Programmatic Assessment	Appropriate assessments and documentation created for clinical encounters, AA meetings & CC meetings; faculty and trainees utilize appropriate forms to prepare for meetings.	Tracking EPAs and assessment completion is a challenge.
EPA descriptions are straightforward and facilitate more timely feedback and documentation; faculty report challenges using forms and understanding how stage-specific entrustment anchors should be used.
There are discordant views on pre-populating assessments.
AA meetings are valuable as a check-in/form of mentorship.
The EC is critical to the CC process (building summary reports).
CC members feel as though they have a strong CC process, although trainees are fairly unaware of the CC processes.	Simplified wording of milestones.
EPA tracking sheets created.
Stage/rotation assessment plan ‘cheat sheets.’
One-page summary of off-service EPA expectations created.
Entrustment scales adjusted where appropriate (i.e. O-SCORE for procedures).
Faculty and trainees agreed there should be a balance between who triggers assessments.
Assessment summary reports for CC members will continue to be refined.
CC standard operating procedures infographic created.	
Abbreviations: AA, academic advisor; CBD, competence by design; CBME, competency-based medical education; CC, competence committee; EC, educational consultant; EPA, Entrustable professional activity; LMS, learning management system; O-SCORE, Ottawa surgical competency operating room evaluation; PGME, postgraduate medical education.

Given the importance of CQI, these results facilitated adaptations such as assessment plan ‘cheat sheets’ with stage-specific EPAs and mental models, EPA tracking forms, and an infographic to detail CC processes in order to improve transparency. These adaptations were instrumental in preparation for national CBD implementation.

Discussion

This early program evaluation in one PM&R program two years post-CBME implementation helped understand the gaps between planned versus enacted implementation across the core components of CBME,9 with adaptations in mind. Several examples of adaptations are described in table 2, such as EPA ‘cheat sheets’ to facilitate a shared mental model and an infographic to improve CC transparency. This evaluation highlights the benefits and challenges of early implementation, allowing these early results to inform those implementing CBME later, including the national implementation of CBD in PM&R.

Ongoing and sustained professional development efforts are needed to shift the culture in a program.4 This is evidenced by the finding that CBME feels like a reductionist approach to learning, which is common in early CBME evaluations,6,7,21 yet is in tension with a growth-mindset.22 Recent evidence suggests that CBME has inadvertently added to trainee assessment burden, thus worsening resident well-being.23 Iterative improvements must balance the benefits and risks of assessment with trainee autonomy and well-being in order for the promise of CBME to be fully actualized. Future PM&R program-level adaptations include an annual education retreat to foster ongoing stakeholder engagement in CQI, and integrating professional development to shift to a growth-oriented assessment system. Such efforts to foster culture change can be implemented in any program, no matter the context.

We acknowledge that this program evaluation is limited to one program at one site, and that this data is several years old. However, we expect that these early implementation findings and adaptations may be helpful to other programs undergoing similar change processes in early stages of CBME implementation. These findings also highlight the importance of ongoing, iterative program evaluation. Next steps include revisiting the outcomes of the adaptations made.

Conclusions

Early program evaluation can demonstrate how actual outcomes do not always align with intended outcomes. Shifting residency program culture takes time and requires sustained efforts in faculty and trainee development.4 Culture change must focus not only on skills and cognitions, but also on the context and the collaborative efforts of all stakeholders.24,25 It is inevitable that aspects of CBME implementation may not achieve the desired outcomes, therefore iterative adaptations and professional development must be responsive so that CBME programs match their intended purpose.26

Conflicts of Interest

The authors have no conflicts of interest to declare.

Edited by

Sarah Burm (section editor); Cindy Schmidt (senior section editor); Marcel D’Eon (editor-in-chief)
==== Refs
1 van Melle E, Hall AK, Schumacher DJ, et al . Capturing outcomes of competency-based medical education: the call and the challenge. Med Teach. 2021;43 (7 ). 10.1080/0142159X.2021.1925640
2 Hamza DM, Ross S, Oandasan I. Process and outcome evaluation of a CBME intervention guided by program theory. J Eval Clin Pract. Vol 26 ; 2020. 10.1111/jep.13344
3 King JA, Morris LL, Fitz-Gibbon CT. How to assess program implementation.; 1987.
4 Hall AK, Rich J, Dagnone JD, et al . It’s a marathon, not a sprint: rapid evaluation of competency-based medical education program implementation. Acad Med. Published online 2020. 10.1097/ACM.0000000000003040
5 Kouzmina E, Mann S, Chaplin T, Zevin B. An evaluation of the surgical foundations curriculum: a national study. J Surg Educ. 2021;78 (3 ). 10.1016/j.jsurg.2020.10.002
6 Federation des Medecins Residents du Quebec. Year 3 of implementation of competence by design: negative impact still outweighs theoretical benefits. observations on the day-to-day reality of CBD and its progression since July 2017. July 2020. Available from: https://fmrq.qc.ca/wp-content/uploads/2022/07/fmrq-report-cbd-implementation-year-3_1.pdf. [Accessed Oct 28, 2023].
7 Sherbino J, Regehr G, Dore K, Ginsburg S. Tensions in describing competency-based medical education: a study of Canadian key opinion leaders. AdvHealth Sci Ed. 2021;26 (4 ). 10.1007/s10459-021-10049-8
8 Gold M, Helms D, Guterman S. Identifying, monitoring, and assessing promising innovations: using evaluation to support rapid-cycle change. Issue Brief (Commonw Fund). 2011;12 .
9 van Melle E, Frank JR, Holmboe ES, Dagnone D, Stockley D, Sherbino J. A core components framework for evaluating implementation of competency-based medical education programs. Acad Med. 2019;94 (7 ):1002-1009. 10.1097/ACM.0000000000002743 30973365
10 Stockley D, Egan R, van Wylick R, et al . A systems approach for institutional CBME adoption at Queen’s University. Med Teach. 2020;42 (8 ):916-921. 10.1080/0142159X.2020.1767768 32486873
11 Frank JR, Snell L, Sherbino J Editors. CanMEDS 2015 Physician Competency Framework. Ottawa: Royal College of Physicians and Surgeons of Canada. CanMEDS 2015 Physician Competency Framework Ottawa: Royal College of Physicians and Surgeons of Canada. Published online 2015.
12 Royal College of Physicians and Surgeons of Canada. What is CBD? The Royal College of Physicians and Surgeons of Canada. https://www.royalcollege.ca/rcsite/cbd/what-is-cbd-e [Accessed Jan 19, 2023].
13 Royal College of Physicians and Surgeons of Canada. The Competence by Design (CBD) coaching model. 2017. Available from: https://www.royalcollege.ca/content/dam/documents/accreditation/competence-by-design/directory/cbd-coaching-model-handout-en.pdf [Accessed Oct 28, 2023].
14 Pero R, Pero E, Marcotte L, Dagnone JD. Educational consultants: fostering an innovative implementation of competency-based medical education. Med Educ. 2019;53 (5 ). 10.1111/medu.13868
15 van Melle E. How do we know it works? (Closing Plenary) Presented at: World Summit on CBME. Published online August 25, 2018.
16 Lipscomb M. Abductive reasoning and qualitative research. Nurs Philos. 2012;13 (4 ). 10.1111/j.1466-769X.2011.00532.x
17 Timmermans S, Tavory I. Theory construction in qualitative research: from grounded theory to abductive analysis. Sociol Theory. 2012;30 (3 ). 10.1177/0735275112457914
18 Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15 (9 ). 10.1177/1049732305276687
19 Birt L, Scott S, Cavers D, Campbell C, Walter F. Member checking: a tool to enhance trustworthiness or merely a nod to validation? Qual Health Res. 2016;26 (13 ). 10.1177/1049732316654870
20 Bloch M, Lave J, Wenger E. Situated learning: legitimate peripheral participation. Man. 1994;29 (2 ). 10.2307/2804509
21 Frank JR, Snell LS, Cate O ten, et al . Competency-based medical education: theory to practice. Med Teach. 2010;32 (8 ). 10.3109/0142159X.2010.501190
22 Richardson D, Kinnear B, Hauer KE, et al . Growth mindset in competency-based medical education. Med Teach. 2021;43 (7 ):751-757. 10.1080/0142159X.2021.1928036 34410891
23 Ott M, Pack R, Cristancho S, et al . “The most crushing thing”: understanding resident assessment burden in a competency-based curriculum. JGME. 2022;14 (5 ). 10.4300/JGME-D-22-00050.1
24 Spillane JP, Reiser BJ, Reimer T. Policy implementation and cognition: Reframing and refocusing implementation research. Rev Educ Res. 2002;72 (3 ). 10.3102/00346543072003387
25 Griffiths J, Dalgarno N, Schultz K, Han H, van Melle E. Competency-Based Medical Education implementation: Are we transforming the culture of assessment? Med Teach. 2019;41 (7 ). 10.1080/0142159X.2019.1584276
26 Hall AK, Woods R, Frank JR. Changing the culture of residency training through faculty development–ERRATUM. CJEM. 2019;21 (6 ). 10.1017/cem.2019.419
