Ophthalmology Program Director Perspectives of Scoring Step 1 Pass/Fail

Abstract

Background Scoring for the United States Medical Licensing Examination (USMLE) Step 1 was recently announced to be reported as binary as early as 2022. The general perception among program directors (PDs) in all specialties has largely been negative, but the perspective within ophthalmology remains uncharacterized.

Objective This article characterizes ophthalmology residency PDs' perspectives regarding the impact of pass/fail USMLE Step 1 scoring on the residency application process.

Methods A validated 19-item anonymous survey was electronically distributed to 111 PDs of Accreditation Council for Graduate Medical Education-accredited ophthalmology training programs.

Results Fifty-six PDs (50.5%) completed the survey. The median age of respondents was 48 years and the majority were male (71.4%); the average tenure as PD was 7.1 years. Only 6 (10.7%) PDs reported the change of the USMLE Step 1 to pass/fail was a good idea. Most PDs (92.9%) indicated that this will make it more difficult to objectively compare applicants, and many (69.6%) did not agree that the change would improve medical student well-being. The majority (82.1%) indicated that there will be an increased emphasis on Step 2 Clinical Knowledge (CK) scores, and many (70.4%) felt that medical school reputation will be more important in application decisions.

Conclusion Most ophthalmology PDs who responded to the survey do not support binary Step 1 scoring. Many raised concerns regarding shifted overemphasis on Step 2 CK, uncertain impact on student well-being, and potential to disadvantage certain groups of medical students including international medical graduates. These concerns highlight the need for reform in the ophthalmology application process.

Keywords

medical education - residency selection - SF match - match process - USMLE - Step 1 - program directors

Ophthalmology is among the most competitive specialties, as each year applicants outnumber the positions available, resulting in a match rate between 72 and 78% since 2011.[1] In turn, United States Medical Licensing Examination (UMSLE) Step 1 scores for ophthalmology applicants are higher on average compared with other specialties and have increased nearly one standard deviation over the past 10 years.[1] [2] To be competitive, students apply to more programs each year and this trend has resulted in an increase in the average number of applications per applicant from 52 to 77 since 2011.[1] This increase in applicants has presented ophthalmology program directors (PDs) with several challenges in selecting candidates and has led to a greater reliance on objective measures, such as grades, class rank, and USMLE scores.[3] In a 2017 survey of ophthalmology PDs, the UMSLE Step 1 score was found to be among the most important factors for screening applicants for interview.[4]

In February 2020, it was announced that USMLE Step 1 score reporting will change from the traditional three-digit numeric score to reporting only a pass/fail outcome as soon as January 2022. A national survey of PDs from across many specialties highlighted the generally negative reactions to the updated scoring system.[5] Yet, the viewpoint of ophthalmology PDs remains uncharacterized. This study aims to clarify the perception among ophthalmology PDs of binary USMLE Step 1 scoring.

Methods

To characterize how ophthalmology PDs will respond to pass/fail USMLE Step 1 scoring, a 19-item survey was validated through phases of prepilot and pilot testing. The survey was iteratively revised, and internal validity was assessed by calculation of the Cronbach's α (0.87). Institutional Review Board/Ethics Committee ruled that approval was not required for this study.

Directors of Accreditation Council for Graduate Medical Education (ACGME)-accredited residency programs in 30 specialties were invited to participate, including 111 ophthalmology PDs.[5] Email addresses were obtained from publicly available ACGME documents. Over the course of 4 weeks, individualized requests for participation were sent through the REDCap (Research Electronic Data Capture) platform.

Descriptive statistics were computed using Microsoft Excel. All values are listed as discrete numbers or percentages with 95% confidence intervals. Statistical significance in response plurality was determined by nonoverlapping 95% confidence intervals.

Results

A total of 56 ophthalmology PDs participated in the survey, representing 44.8% of the ACGME-accredited residency programs. The majority of responses were evenly distributed among programs from the Northeast (32.1%), South (26.8%), West (14.3%), and Midwest (26.8%). The median PD age was 48 years and the majority were male (71.4%); the average tenure as PD was 7.1 years ([Table 1]). This sex distribution and average tenure among the ophthalmology PDs was consistent with the characteristics of respondents from all specialties.[5]

Table 1 Demographic characteristics of ophthalmology residency program director survey respondents

Characteristics	Participants, no. (%)
Age, y, median (IQR)	48 (41–58)
Male gender	40 (71.4)
Tenure in position, y, median (IQR)	5 (3–10)
Region
Northeast	18 (32.1)
South	15 (26.8)
Midwest	15 (26.8)
West	8 (14.3)

Only 10.7% of PD respondents felt that changing the USMLE Step 1 to pass/fail was a good idea, and the majority (92.9%) anticipate it will become more difficult to objectively compare applicants. Furthermore, the elimination of numerical Step 1 reporting will increase their emphasis on the USMLE Step 2 Clinical Knowledge (CK) scores, and 76.9% of PDs will require applicants to submit Step 2 CK scores at the time of application. Responses indicate that this shift in reporting will likely lead to increased emphasis on an applicant's medical school reputation during the application process (70.4%) and negatively affect international medical graduates (73.2%). When asked if the change to pass/fail grading would improve student well-being, the majority of PDs were neutral (46.4%). Lastly, only 9.1% supported a similar change to pass/fail for Step 2 CK scoring reports ([Table 2]).

Table 2 Ophthalmology program director perspectives of scoring Step 1 pass/fail

Statement	Disagree	Neutral	Agree	Summary of responses from all specialties[5]
	Percent (95% confidence interval)
Changing the USMLE Step 1 to pass/fail:
Is a good idea	66.1 (53.7–78.5)[a]	23.2 (12.2–34.3)	10.7 (02.6–18.8)	Disagree – 60.8 (58.8–63.6)[a]
Will make it more difficult to objectively compare applicants	5.4 (0.0–11.3)	1.8 (0.0–5.3)	92.9 (86.1–99.6)[a]	Agree – 77.2 (74.8–79.6)[a]
Will increase emphasis on Step 2 CK scores in selecting applicants for my program	5.4 (0.0–11.3)	12.5 (3.8–21.2)	82.1 (72.1–92.2)[a]	Agree – 80.7 (78.4–82.0)[a]
Will put international medical graduates at a disadvantage	12.5 (3.8–21.2)	14.3 (5.1–23.5)	73.2 (61.6–84.8)[a]	Agree – 44.4 (41.6–47.3)[a]
Will decrease socioeconomic disparities in the application process	42.9 (29.9–55.8)	42.9 (29.9–55.8)	14.3 (5.1–23.5)	Nonsignificant
Will decrease medical student knowledge of the basic sciences	23.2 (12.2–34.3)	46.4 (33.4–59.5)	30.4 (18.3–42.4)	Nonsignificant
Will improve medical student well-being	23.2 (12.2–34.3)	46.4 (33.4–59.5)	30.4 (18.3–42.4)	Nonsignificant
Will make applicant screening more arduous	3.6 (0.0–8.4)	1.8 (0.0–5.3)	94.6 (88.7–100.0)[a]	–
As a result of changing USMLE Step 1 to pass/fail:
I will now require applicants to submit Step 2 CK scores with ERAS/CAS	7.7 (0.4–14.9)	15.4 (5.6–25.2)	76.9 (65.5–88.4)[a]	Agree – 77.1 (74.7–79.5)[a]
Where an applicant goes to medical school will be more important in screening and selection for my program	13.0 (4.0–21.9)	16.7 (6.7–26.6)	70.4 (58.2–82.5)[a]	Agree – 56.8 (54.0–59.7)[a]
Step 2 CK should also be changed to pass/fail	80.0 (69.4–90.6)[a]	10.9 (2.7–19.1)	9.1 (1.5–16.7)	Disagree – 60.8 (58.8–63.6)[a]

Abbreviations: CAS, Central Application Service; CK, Clinical Knowledge; ERAS, Electronic Residency Application Service; USMLE, United States Medical Licensing Examination.

^a Indicates a statistically significant plurality of responses by nonoverlapping confidence intervals.

Discussion

In this analysis, ophthalmology PDs generally disapproved of the change to pass/fail Step 1 scoring. The USMLE Step 1 exam is the first of a three-part exam series originally established for medical licensure decisions but over time has evolved to include many secondary functions, including selection of residency applicants and honor society members.[6] The USMLE scoring reform was intended to improve the transition from undergraduate medical education to graduate medical education while balancing student learning and well-being.[7] Specifically, the change was aimed at reducing the current perceived overemphasis on USMLE performance, which has become particularly prevalent in competitive specialties such as ophthalmology. However, ophthalmology PDs are skeptical of the predicted benefits with several concerns raised regarding possible unintended consequences of this isolated change.

Another impetus for change was to address the widening of socioeconomic and racial disparities in the application process. Step 1 scores have been found to vary by race, gender, and socioeconomic status.[8] While removing Step 1 scores may eliminate some of the disparities associated with the exam, PDs were still overall undecided about the impact on socioeconomic disparities in the application process. This may be because eliminating numerical Step 1 scores will simply lead to increased focus on Step 2 CK, a test with similar problems.[9] Yet, most PDs disagreed (80.0%) with implementing a similar scoring change to the Step 2 CK exam, highlighting a need for objective measures. With most PDs (70.4%) indicating that binary Step 1 scoring will result in more weight being placed on medical school reputation, the socioeconomic disparities may remain unchanged and international medical graduates may be negatively affected. Top medical schools and institutions with greater National Institutes of Health funding are more likely to have resources and mentors for students to engage in research, a widely known metric in the screening and review process that may become more important after the scoring change.[3] Students at these institutions are also more likely to have an ophthalmology department and access to well-known faculty within the specialty. This increased reliance on medical school reputation may adversely impact attempts to improve diversity within the specialty.

While the negative perceptions of a pass/fail Step 1 are not unique to the specialty,[5] the ophthalmology PDs represented much more concordant and decisive opinions when compared with the general perspective of all specialties ([Table 2]). This is no surprise as the ophthalmology match process is positioned to be more drastically affected due to the early match timeline. Applicants for ophthalmology begin registration in early summer and typically submit applications by mid-August. As a result of this timeline, Step 2 CK scores are usually not available in time for the screening and selection process. In fact, only 46% of ophthalmology applicants in 2011 reported their Step 2 score.[10] Because of the increased emphasis on Step 2 CK, medical students may need to complete this exam before the conclusion of their third year. This shortened timeframe may place a financial burden on students from lower socioeconomic backgrounds as they will now be required to pay registration fees (over $1,000 combined) for both exams within the same year. Students will also feel more pressure to perform well on Step 2 CK knowing that there will no longer be a “second chance” to redeem initial performance. This academic stress may also disrupt clinical rotations and other curricular programs as students prepare for Step 2 CK.

Current literature regarding the use of Step 2 CK as a residency selection tool and predictor of resident performance is scarce—a concerted effort needs to be dedicated to interpreting and understanding these results in the context of ophthalmology training. Additional consideration should be focused on possible reforms to the match process that will improve applicants' ability to present their best attributes while promoting diversity and inclusion within the specialty. Though progress has been made toward this goal, no single clear method has emerged.[11] Further research is necessary to aid PDs and admission committees to find creative, evidence-based methods to evaluate applicants holistically. While the intention of the Step 1 reform was to promote a more comprehensive review of applicants beyond standardized test performance, an unintentional consequence may be that more emphasis is placed on factors that further disadvantage groups already underrepresented in the field.

Limitations of this study include possible self-selection bias as PDs with more opinionated views on the topic may be more likely to respond to the online survey. While the response rate and sample size may limit generalizability, this study still provides the largest sample size (to the authors' knowledge) of responses from PDs with regard to the new Step 1 scoring change. Future investigations should assess additional characteristics of PDs and their respective programs to understand the correlation between survey responses and factors such as research funding and activity, program size, and ranking success.

Conclusion

In summary, the majority of ophthalmology PD respondents do not support pass/fail Step 1 scoring, citing the lack of benefit on student well-being, shifted emphasis on Step 2 CK, and greater burden of residency selection. An isolated change to Step 1 scores without concurrent reform of the application may be detrimental to those involved in the ophthalmology early match process.

Conflict of Interest

None declared.

• References

• 1 Match SF. Ophthalmology Residency Match Summary Report. 2019 . Available at: https://www.sfmatch.org/PDFFilesDisplay/Ophthalmology_Residency_Stats_2019.pdf. Accessed May 5, 2020
• 2 National Resident Matching Program, Characteristics of U.S. Allopathic Seniors Who Matched to Their Preferred Specialty in the 2018 Main Residency Match. October 2019
• 3 Nallasamy S, Uhler T, Nallasamy N, Tapino PJ, Volpe NJ. Ophthalmology resident selection: current trends in selection criteria and improving the process. Ophthalmology 2010; 117 (05) 1041-1047
• 4 Behunin N, Hill C, Lotpifour M, Scott IU. The ophthalmology residency program director survey: applicant qualities affecting the invitation to interview and rank position. Invest Ophthalmol Vis Sci 2017; 58 (08) 5052
• 5 Makhoul AT, Pontell ME, Ganesh Kumar N, Drolet BC. Objective measures needed - program directors' perspectives on a pass/fail USMLE Step 1. N Engl J Med 2020; 382 (25) 2389-2392
• 6 Andolsek KM. One small step for Step 1. Acad Med 2019; 94 (03) 309-313
• 7 Summary Report and Preliminary Recommendations from the Invitational Conference on USMLE Scoring (InCUS), March 11–12; 2019, United States Medical Licensing Examination. Accessed March 29, 2020
• 8 Rubright JD, Jodoin M, Barone MA. Examining demographics, prior academic performance, and United States Medical Licensing Examination scores. Acad Med 2019; 94 (03) 364-370
• 9 Gauer JL, Jackson JB. Relationships of demographic variables to USMLE physician licensing exam scores: a statistical analysis on five years of medical student data. Adv Med Educ Pract 2018; 9: 39-44
• 10 Yousuf SJ, Jones LS. Ophthalmology residency match outcomes for 2011. Ophthalmology 2012; 119 (03) 642-646
• 11 Lee AG, Golnik KC, Oetting TA. et al. Re-engineering the resident applicant selection process in ophthalmology: a literature review and recommendations for improvement. Surv Ophthalmol 2008; 53 (02) 164-176

Publication History

Received: 23 June 2020

Accepted: 03 September 2020

Article published online:
26 November 2020