
==== Front
Med Educ Online
Med Educ Online
Medical Education Online
1087-2981
Taylor & Francis

39244774
10.1080/10872981.2024.2396166
2396166
Version of Record
Rapid Communication
Equity, Diversity and Inclusion in Medical Education
Recommendations to address and research systemic bias in assessment: perspectives from directors of research in medical education
F. CHEN ET AL.
MEDICAL EDUCATION ONLINE
Chen Fei a
O’Brien Celia Laird b
Blanco Maria A. c
Huggett Kathryn N. d
Jeffe Donna B. e
Pusic Martin V. f
Brenner Judith M. g
a Department of Anesthesiology, University of North Carolina , Chapel Hill, NC, USA
b Department of Medical Education, Northwestern University Feinberg School of Medicine , Chicago, IL, USA
c Office of Educational Affairs, Tufts University School of Medicine , Boston, MA, USA
d Teaching Academy, Larner College of Medicine at the University of Vermont , Burlington, VT, USA
e Department of Medicine, Washington University in St. Louis School of Medicine , St. Louis, MO, USA
f Department of Pediatrics, Harvard Medical School , Boston, MA, USA
g Department of Medicine, New York University Grossman Long Island School of Medicine , Mineola, NY, USA
CONTACT Fei Chen fei_chen@med.unc.edu Department of Anesthesiology, University of North Carolina, N2198 UNC Hospitals CB#7010, Chapel Hill, NC 27599-7010, USA
8 9 2024
2024
8 9 2024
29 1 2396166Integra06 9 2024
Integra06 9 2024
04 12 2023
18 3 2024
20 8 2024
© 2024 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.
2024
The Author(s)
https://creativecommons.org/licenses/by-nc/4.0/ This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. The terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent.

ABSTRACT

Introduction

Addressing systemic bias in medical school assessment is an urgent task for medical education. This paper outlines recommendations on topic areas for further research on systemic bias, developed from a workshop discussion at the 2023 annual meeting of the Society of Directors of Research in Medical Education.

Materials and Methods

During the workshop, directors engaged in small-group discussions on guidelines to address bias in assessment practices following a proposed categorization of ‘Do’s,’ ‘Don’ts,’ and ‘Don’t knows’ and listed their insights using anonymous sticky notes, which were shared and discussed with the larger group of participants. The authors performed a content analysis of the notes through deductive and inductive coding. We reviewed and discussed our analysis to reach consensus.

Results

The workshop included 31 participants from 28 institutions across the US and Canada, generating 51 unique notes. Participants identified 23 research areas in need of further study. The inductive analysis of proposed research areas revealed four main topics: 1) The role of interventions, including pre-medical academic interventions, medical-education interventions, assessment approaches, and wellness interventions; 2) Professional development, including the definition and assessment of professionalism and professional identity formation; 3) Context, including patient care and systemic influences; and 4) Research approaches.

Discussion

While limited to data from a single workshop, the results offered perspectives about areas for further research shared by a group of directors of medical education research units from diverse backgrounds. The workshop produced valuable insights into the need for more evidence-based interventions that promote more equitable assessment practices grounded in real-world situations and that attenuate the effects of bias.

KEYWORDS

Medical education
assessment equity
systemic bias
evidence-based
research recommendations
The author(s) reported that there is no funding associated with the work featured in this article.
==== Body
pmcIntroduction

Bias can critically impact assessment with adverse repercussions on learner development and growth. Therefore, addressing systemic bias in medical school assessment is a task of urgency and importance for medical education [1–4]. Medical education leaders have proposed several recommendations to foster equity in assessment practices, many of which require a systematic study to show effectiveness [5–7].

While awareness of inequities in medical education outcomes has increased, much remains unknown about effective ways to mitigate bias in assessment [8]. Identifying the most urgent topics of study needs to be a priority of the medical education community.

This paper identifies salient research topics to foster equitable assessments proposed by members of the Society of Directors of Research in Medical Education (SDRME) at the 2023 annual meeting. SDRME includes a diverse set of medical education scholars from institutions both within and outside of the US.

Materials and methods

During the annual SDRME meeting, we hosted a workshop to facilitate small-group discussions about experiences with and perspectives on bias in institutional assessment practices among participants. The workshop was inspired by Fyfe et al.’s work, which reviewed research evidence on differential attainment related to race/ethnicity in medical schools [8]. Those authors proposed guidelines to enact changes in assessment practices and identify areas for further research, including practices for which there is 1) evidence of having positive impact (Do’s), 2) evidence of having negative effects or no effectiveness (Don’ts), and 3) a lack of evidence of effectiveness and need for further research (Don’t knows).

In small-group discussions at the workshop, the participants discussed practices at their schools in the context of the ‘Do’s, Don’ts, and Don’t knows.’ For the discussion of the ‘Don’t knows,’ we asked participants to generate topics in need of further research (see Appendix). Each group recorded their discussion points anonymously on sticky notes, which were posted on a wall of the meeting room and shared their points with the larger group.

Following the conference, we collated all the notes and conducted a conventional content analysis using both deductive and inductive analytic approaches [9]. Three authors (FC, CLO, MAB) first independently coded the notes, then deductively mapped each note to Fyfe et al.’s categories of ‘Do’s, Don’ts, and Don’t knows’, as applicable, and finally inductively coded the notes to identify new topics. We then met to discuss and reconcile disagreements. To ensure trustworthiness, we included consensus coding through an iterative process and maintained an audit trail throughout the analysis [9,10]. Finally, we reviewed, discussed and further refined our proposed topics and sub-topics with the rest of the authors.

The Institutional Review Board at the University of North Carolina determined this project to be non-human-subjects research.

Results

Thirty-one participants from 28 institutions in the US and Canada attended the in-person SDRME meeting. Participants identified 23 research areas in need of further study. The analysis of these research areas is summarized in Table 1. We identified four main topics: 1) The role of interventions, including pre-medical academic interventions, medical-education interventions, assessment approaches, and wellness interventions; 2) Professional development, including the definition and assessment of professionalism and professional identity formation; 3) Context, including patient care and systemic influences; and 4) Research approaches. Table 1 includes descriptions of the topics with examples.Table 1. Participants’ proposed research needs, by topics and guidelines.

Topic	Description	Subtopic	Examples	Related guidelines of “Don’t Knows” by Fyfe et al.a	
The role of interventions	These research questions focused on studying interventions meant to support medical students’ learning and wellness and reduce the impact of racism and bias.	Medical-education interventions	Can interventions that help with systemic issues like stereotype threat help improve outcomes?

Are “level up” programs effective for curriculum beyond foundational sciences? How? For whom?

	1, 3	
Pre-medical academic interventions	Develop more pipeline and pathways for learners.

Do pre-matriculation students experience imposter syndrome differently from their peers?

	1, 3	
Assessment approaches	What is the role of longitudinal learner profiles in improving attainment? For whom? How?

Assessment of competency: what works?

	3	
Wellness interventions	What are outcomes of well-being programs?

[What is the] impact of giving every student a “freedom day”?

	1	
Professional Development	These research questions focused on exploring how we define professionalism and how we assess professionalism and professional identity development.	Professionalism	What is the definition of professionalism and is it associated with certain groups? a. Are we using “professionalism” definitions inappropriately? b. Are we using “professionalism” definitions consistently across all groups? c. How do “unprofessional” behaviors affect patient care?

Assessment of professionalism through DEI lens.

	2	
Professional Identity	How do we assess emerging of professional identity and what are the issues that this raises?

	2	
Context	These research questions focused on studying the larger context of assessment in medical education, including its impact on patient care.	Patient Care	How does all of this [efforts to improve equity in assessment] improve patient care, safety and outcomes?

How to create a safe space for patients including anti-racism instrument?

	Not applicable	
Systematic influences	Detecting bias [is a] political act—we have to understand the context in which we are operating

	Not applicable	
Research Approaches	These research questions focused on the research process itself.	 	What are methodologies that can detect bias?

What are some explanatory models for differential attainment?

	Not applicable	
aKey to Fyfe’s categories of “Don’t Knows”: 1. Are supportive interventions for specific groups effective in respect of assessment outcomes? 2. To what extent do professionalism norms and assessments of professionalism discriminate against minoritized students? 3. How can formal curricula influence differential attainment?

The majority of the questions focused on the role of interventions as well as on the definition and assessment of professionalism; both of these topics are reflected in the ‘Don’t Knows’ from Fyfe et al. The participants also highlighted two other areas in need of further research that are not covered in the guidelines: the larger context of assessment, including its impact on patient care, and the research processes that are used to study assessment practices.

Discussion

Our analysis of the workshop notes revealed that the participants identified ways to operationalize Fyfe et al.’s proposed guidelines while also suggesting new areas in need for further research. For example, the participants voiced a need to employ appropriate research approaches to study equitable assessments. Studying assessment inequity is challenging due to the many confounding factors that are involved in the process, including structural and personal biases and educational disparities that begin long before medical school [2,6]. As the participants recommended, accounting for the context of assessments is critical while conducting research that aims at addressing systemic bias in assessment. This indicates alignment with those arguing for systematic approaches to eliminating assessment biases as well as those highlighting how these inequities lead to subpar patient care [5–7]. The results included issues that are beyond the scope of assessments per se but are intrinsically associated with it, such as wellness interventions and the culture and context of medical education.

We acknowledge the limitations that our data came from a single workshop with a self-selected sample of participants and that the data were collected anonymously; demographic characteristics were neither requested nor provided. Still, the perspectives shared by this group of directors of medical education research units from diverse social/cultural and training backgrounds revealed insights grounded in real-world situations and suggest consensus that these research topics should be studied to reduce disparities in assessment due to systemic bias. It is our hope that this paper contributes to the ongoing dialogue about interventions that foster equitable assessment practices and enhance the quality of medical education, specifically through research. Next steps will include sharing our findings with the larger society membership to solicit recommendations for action. Recommendations might include specific research projects, which could arise from one or more examples listed for the topics shown in Table 1 and aim to promote equity and to reduce disparities in assessments due to bias.

Acknowledgments

The authors would like to thank the attendees of Society of Directors of Medical Education 2023 summer conference for their participation in the workshop.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Appendix. Small group session instructions

Pick 2-3 ‘Do’s’ and 2-3 ‘Don’ts’

Discuss the following questions How are the ‘Do’s’ operationalized at your institutions?

Were any of these ‘Don’ts’ occurring at your institution?

How did your institution successfully transform into recommended/evidence-informed practices?

If there were no observations of the ‘Don’ts’ in your institutions’ practices, what could be done as preventative practices to avoid the occurrence of these undesirable practices?

The Don’t Knows Ideas for research questions?

Approaches to advance our understanding?

Any other important ‘Don’t knows’ missing here?
==== Refs
References

[1] Josiah M Jr. Foundation conference on ensuring fairness in medical education assessment: conference recommendations report. Acad Med. 2023 Aug 1;98(8S):S3–S15.
[2] Teherani A, Hauer KE, Fernandez A, et al. How small differences in assessed clinical performance amplify to large differences in grades and awards: a cascade with serious consequences for students underrepresented in medicine. Acad Med. 2018;93 (9 ):1286–4.29923892
[3] Henderek JJ, Tong Y, Rubright JD, et al. The inaugural equity in measurement and assessment conference. (EMAC): NBME; 2023.
[4] Low D, Pollack SW, Liao ZC, et al. Racial/Ethnic disparities in clinical grading in medical school. Teach Learn Med. 2019;31 (5 ):487–496. doi: 10.1080/10401334.2019.1597724 31032666
[5] Hauer KE, Park YS, Bullock JL, et al. “My assessments are biased!” measurement and sociocultural approaches to achieve fairness in assessment in medical education. Acad Med. 2023 Aug 1;98 1 :S16–S27.37094278
[6] Anderson N, Nguyen M, Marcotte K, et al. The long shadow: a historical perspective on racism in medical education. Acad Med. 2023 Aug 1;98 (8S) :S28–S36.
[7] Lucey CR, Hauer KE, Boatright D, et al. Medical Education’s wicked problem: achieving equity in assessment for medical learners. Acad Med. 2020;95 (12S Addressing Harmful Bias and Eliminating Discrimination in Health Professions Learning Environments) S98–S108.32889943
[8] Fyfe M, Horsburgh J, Blitz J, et al. The do’s, don’ts and don’t knows of redressing differential attainment related to race/ethnicity in medical schools. Perspect Med Educ. 2022;11 (1 ):1–14.34964930
[9] Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15 (9 ):1277–1288.16204405
[10] Johnson JL, Adkins D, Chauvin S. A review of the quality indicators of rigor in qualitative research. Am J Pharm Educ. 2020;84 (1 ):7120.32292186
