Keywords
OKAP - Ophthalmic Knowledge Assessment Program - face-to-face lectures - traditional
didactic lecture - resident education - synchronous learning - asynchronous learning
- didactic lecture
With an increased number of learning platforms and continual changes in the way residents
learn, the relative role of traditional lectures in conveying ophthalmic knowledge
is of interest. Faculty members put many hours into preparing and updating lecture
content, which becomes rapidly outdated. Therefore, optimization in producing learning
content is of value.
Studies assessing the relative efficacy of problem-based learning versus lecture-based
learning have shown varying results for short-term and longer-term learning.[1]
[2] The contemporary learner has been exposed to question banks, preparatory courses
for the United States Medical Licensing Exams, and a variety of tools from online
resources to simulation laboratories. With an ever-increasing quantity of knowledge
and skills required to practice ophthalmology and a fixed term of learning, it is
worthwhile to determine what is most efficient in transferring medical knowledge and
skills. In ophthalmology, this knowledge may be estimated with the Ophthalmic Knowledge
Assessment Program (OKAP). It is a 260-item multiple-choice test that is administered
to ophthalmology residents in each year of training. It is designed to measure the
ophthalmic knowledge of residents, relative to their peers, to facilitate the ongoing
assessment of resident progress and program effectiveness. Residents are assessed
relative to their peers in 13 categories plus the overall average performance. This
is expressed as a percentile rank relative to the training year peers.
In this study, we attempt to estimate the value per hour of traditional slide-based
face-to-face lectures with respect to acquisition of knowledge using the OKAP's overall
average percentile score as the proxy.
Purpose
To assess the efficacy of traditional slide-based face-to-face didactic lectures on
ophthalmic knowledge acquisition as estimated through the average OKAP performance.
Methods
This study was reviewed by the Institutional Review Board (IRB) at the University
of Arizona and IRB oversight was deemed unnecessary. We reviewed yearly records of
didactic attendance from 2011 through 2019. This corresponded with 8 years of OKAP
test results available for 12 residents per year with a total of 40 residents being
represented. The sum of total hours, from immediately after an OKAP examination until
immediately before the following exam, were recorded per resident along with their
corresponding average percentage OKAP scores. All lectures during this period were
of the traditional slide-based face-to-face lecture format.
A simple regression analysis was used to determine if a linear relationship existed
between average OKAP scores and the total of yearly didactic hours per resident. This
relationship was also subsequently subanalyzed by postgraduate year (PGY).
Finally, the effects of the categorical variable, PGY level, on the OKAP performance
to lecture hour relationship were assessed using one-way analysis of variance (ANOVA)
and pairwise comparisons using the Tukey's post hoc test. The ANOVA and Tukey's test
were also performed to determine PGY associations with the number of didactic hours
as well as the OKAP scores. STATA 13.1 IC (Stata Corp, College Station, TX) was used
for all statistical analysis.
Results
Using a linear regression model of average OKAP scores and yearly total hours of didactics
there was a statistically significant positive slope of 0.285%/hour of instruction
per year (95% confidence interval [CI] 0.08–0.49) ([Table 1]). Restated, for every 3.5 hours (95% CI 2–13 hours) of didactics a 1% increase in
the average OKAP score was found. The low level of correlation is demonstrated by
the adjusted R
2 of 0.067 ([Table 1]) and by the scatter plot ([Fig. 1]).
Fig. 1 Average Ophthalmic Knowledge Assessment Program (OKAP) score in percent versus yearly
hours of didactic lectures for individual residents from 2012 to 2019. Slope = 0.285,
p = 0.006, R
2 = 0.067.
Table 1
Regression of average OKAP scores against yearly total of didactic hours per resident
|
All PGY[a]
|
SS
|
df
|
MS
|
Number of observation
|
96
|
|
Model
|
6,404
|
1
|
6,404
|
Prob > F
|
0.0064
|
|
Residual
|
77,537
|
94
|
825
|
R
2
|
0.076
|
|
Total
|
83,941
|
95
|
884
|
Adjusted R
2
|
0.067
|
|
OKAP
|
Coefficient
|
Std. Err.
|
t
|
p > |
t|
|
[95% CI]
|
|
Hours
|
0.29
|
0.10
|
2.8
|
0.006
|
0.082, 0.49
|
|
|
Constant
|
9.1
|
15.2
|
0.6
|
0.55
|
–21, 39
|
|
Abbreviations: CI, confidence interval; df, degrees of freedom; MS, mean of sum of
squares; OKAP, Ophthalmic Knowledge Assessment Program; PGY, postgraduate year; SS,
sum of squares; Std. Err., standard error of the mean.
a Represents PGY 2, 3, and 4 residents.
This relationship was driven predominantly by the PGY 4 group which was the only PGY
group, on subgroup analysis, to show a statistically significant slope, 0.36%/hour
of instruction per year, p = 0.0024 ([Table 2]). The PGY 4 group also showed the greatest adjusted R
2 = 0.24 ([Table 2]).
Table 2
Regression of average OKAP scores versus hours of didactic instruction by PGY year
|
PGY 2
|
SS
|
df
|
MS
|
Number of observation
|
32
|
|
Model
|
2,611
|
1
|
2,611
|
Prob > F
|
0.0851
|
|
Residual
|
24,710
|
30
|
824
|
R
2
|
0.096
|
|
Total
|
27,320
|
31
|
881
|
Adjusted
R
2
|
0.065
|
|
OKAP
|
Coefficient
|
Std. Error
|
t
|
p > |t|
|
[95% CI]
|
|
Hours
|
–0.51
|
0.29
|
–1.78
|
0.085
|
–1.1, 0.076
|
|
Constant
|
129
|
41
|
3.12
|
0.004
|
44, 213
|
|
PGY 3
|
SS
|
df
|
MS
|
Number of observation
|
32
|
|
Model
|
1,643
|
1
|
1,643
|
Prob > F
|
0.18
|
|
Residual
|
26,607
|
30
|
887
|
R
2
|
0.058
|
|
Total
|
28,250
|
31
|
911
|
Adjusted
R
2
|
0.027
|
|
OKAP
|
Coefficient
|
Std. Error
|
t
|
p > |t|
|
[95% CI]
|
|
Hours
|
0.32
|
0.23
|
1.36
|
0.184
|
–0.16, 0.80
|
|
Constant
|
7.4
|
38
|
0.2
|
0.847
|
–70, 85
|
|
PGY 4
|
SS
|
df
|
MS
|
Number of observation
|
32
|
|
Model
|
5,394
|
1
|
5,394
|
Prob > F
|
0.0024
|
|
Residual
|
14,743
|
30
|
491
|
R
2
|
0.27
|
|
Total
|
20,138
|
31
|
650
|
Adjusted
R
2
|
0.24
|
|
OKAP
|
Coefficient
|
Std. Error
|
t
|
p > |t|
|
[95% CI]
|
|
Hours
|
0.36
|
0.11
|
3.31
|
0.002
|
0.14, 0.58
|
|
Constant
|
–11
|
15
|
–0.7
|
0.49
|
–41, 20
|
Abbreviations: CI, confidence interval; df, degrees of freedom; MS, mean of sum of
squares; OKAP, Ophthalmic Knowledge Assessment Program; PGY, postgraduate year; SS,
sum of squares; Std. Error, standard error of the mean.
There was a relationship between PGY level and both yearly hours of didactics and
OKAP scores. In the former, PGY 3 residents attended a statistically significant larger
number of didactic hours than both PGY 2 and PGY 4 residents ([Table 3]). PGY 3–PGY 2 = 18 hours, p = 0.025, PGY 3–PGY 4 = 25 hours, p = 0.001, and PGY 4–PGY 2–6.6 hours, p = 0.97.
Table 3
ANOVA of yearly didactic hours by PGY level with pairwise comparison of PGY levels
using Tukey's test
|
ANOVA hours vs. PGY, sequential
|
|
Number of observations = 96
|
R
2 = 0.13
|
|
Root mean square = 27
|
Adjusted R
2 = 0.11
|
|
Seq SS
|
df
|
MS
|
F
|
Prob >
F
|
|
Model
|
10,372
|
2
|
5,186
|
7.03
|
0.0014
|
|
PGY
|
10,372
|
2
|
5,186
|
7.03
|
0.0014
|
|
Residual
|
68,574
|
93
|
737
|
|
|
|
Total
|
78,946
|
95
|
831
|
|
|
|
PGY
|
Margin
|
Std. Error
|
95% unadjusted CI
|
|
2
|
142
|
4.8
|
132, 151
|
|
3
|
160
|
4.8
|
150, 169
|
|
4
|
135
|
4.8
|
126, 145
|
|
PGY
|
Contrast
|
Std. Error
|
Tukey's test t, p > |t|
|
95% CI unadjusted
|
|
3 vs. 2
|
18
|
6.8
|
2.65
|
0.025
|
1.9, 34
|
|
4 vs. 2
|
–6.6
|
6.8
|
–0.97
|
0.600
|
–23, 9.6
|
|
4 vs. 3
|
–25
|
6.8
|
–3.6
|
0.001
|
–41, –8.4
|
Abbreviations: ANOVA, analysis of variance; CI, confidence interval; df, degrees of
freedom; F, Fischer's statistic; MS, mean of sum of squares; PGY, postgraduate year; Seq SS,
sequential sum of squares; Std. Error, standard error of the mean.
Similarly, there was a statistically significant greater performance on the OKAPs
by PGY 3 and PGY 2 relative to PGY 4 residents (PGY 3–PGY 4 21%, p = 0.012, PGY 2–PGY 4 18%, p = 0.034). PGY 2 and 3 were not significantly different, PGY 3–PGY 2 3%, p = 0.92 ([Table 4]).
Table 4
ANOVA of OKAP by PGY level with pairwise comparison of PGY levels using Tukey's test
|
ANOVA OKAP vs. PGY, sequential
|
|
Number of observations = 96
|
R
2 = 0.098
|
|
Root mean square = 29
|
Adjusted R
2 = 0.079
|
|
Seq SS
|
df
|
MS
|
F
|
Prob >
F
|
|
Model
|
8,233
|
2
|
4,117
|
5.06
|
0.0082
|
|
PGY
|
8,233
|
2
|
4,117
|
5.06
|
0.0082
|
|
Residual
|
75,708
|
93
|
814
|
|
|
|
Total
|
83,941
|
95
|
884
|
|
|
|
PGY
|
Margin
|
Std. Error
|
95% unadjusted CI
|
|
2
|
56
|
5
|
46, 66
|
|
3
|
58
|
5
|
48, 68
|
|
4
|
38
|
5
|
28, 48
|
|
PGY
|
Contrast
|
Std. Error
|
Tukey's test
t, p > |t|
|
95% CI
unadjusted
|
|
3 vs. 2
|
2.75
|
7
|
0.39
|
0.92
|
–14, 20
|
|
4 vs. 2
|
–18.1
|
7
|
–2.54
|
0.034
|
–35, –1
|
|
4 vs. 3
|
–20.9
|
7
|
–2.93
|
0.012
|
–38, –4
|
Abbreviations: ANOVA, analysis of variance; CI, confidence interval; df, degrees of
freedom; MS, mean of sum of squares; OKAP, Ophthalmic Knowledge Assessment Program;
PGY, postgraduate year; Seq SS, sequential sum of squares; Std. Error, standard error
of the mean; t, Student's t statistic.
Discussion
There are several cross-sectional studies in the literature that show a statistically
significant positive correlation between the relative quantity of didactic lectures
with medical learning.[3]
[4]
[5] These studies evaluated graduate medical learning in variety of disciplines and
topics.
The study presented here provides additional evidence to suggest that traditional
face-to-face lecture-based didactic sessions have a role in improving medical knowledge.
In this case, there was a statistically significant relationship with the OKAP examinations
but the correlation, as seen with other studies, was low suggesting that there are
other factors playing a significant role in medical education.[3] The PGY 2 and PGY 3 subgroups showed no statistically significant correlation between
the number of didactic lecture hours and OKAP performance. For PGY 4 residents, there
was a statistically significant correlation suggesting that for every 2.8 hours (95%
CI 1.7–7.3 hours) of lecture per year the average OKAP score increased 1%, p = 0.0024. This single factor explained approximately 24% of the variability for this
group's average OKAP performance. For all three PGY levels taken together the R
2 suggested that only 6.7% of the variability was explained by lecture hours and the
slope for all three taken together showed that a greater number of hours were required,
3.5 (95% CI 2–13 hours), to increase the average OKAP performance by 1%. It is possible
that repetition and a greater platform of ophthalmic knowledge in the senior year
make lectures higher yield.
In the present study, there was a statistically significant association between PGY
level and both yearly hours of didactics and OKAP scores. Here, the PGY 3 level showed
the greatest number of didactic hours and the highest OKAP performance. There are
several patterns in the program, from which this study was derived, to explain these
PGY differences. The PGY 3 residents are sent to a multiday out of state review course
approximately 1 week preceding the OKAP exam which increases the amount of dedicated
and optimally timed learning relative to the other two groups. A new environment for
learning with the away course may play a role as well. Also, residents interested
in fellowships are often asked for their OKAP scores thus increasing the incentive
to do well on the PGY 3 year test. Additionally, the PGY 2 residents arrive on July
1 undergoing on-boarding during the first few weeks and effectively obtain 8 months
of didactics as compared with 12 months for the other two groups. Finally, it has
been our experience that the senior residents do not value a high score on the OKAPs
as greatly and find the demands of improving cataract proficiency and job interviews
as competing factors to at-home study. This may also partially explain why mandatory
lectures, thus responsible for a larger percentage of their studies, play a more significant
role in the senior year.
The literature would suggest that complementary or alternative methods to traditional
lecture-based presentations might be effective at improving learning as well as improving
the environment for both teacher and student. Several alternative methods have been
shown to be effective. One example, case-based small discussion groups, in connection
with traditional lectures significantly improved test scores at the conclusion of
two of the five topics investigated. This crossover study involved 170 medical students.[6] Similarly, computer-based learning was found to supplement face-to-face lectures
in a study of 26 medical students learning dermato-oncology. Here, the test results
from questions corresponding the computer-based portion of the curriculum were superior
to those corresponding to the traditional lectures, p < 0.05.[7]
Furthermore, an online text-based learning system was shown to be significantly more
effective, p = 0.012, when an aural accompaniment was present and also lead to a greater number
of students seeking out additional study material.[8] Adding a problem-based component to the didactic curriculum has also proven effective.
In a study with 95 nursing students randomly assigned to traditional lectures, lectures
plus problem-based learning and solely problem-based learning, those with a component
of problem-based learning did significantly better, p = 0.001, on testing immediately after and after weeks of delay.[9] Another creative adjunct to standard anatomy lectures involved the addition of live
narrated interactive video streaming of live surgical procedures. In this nonrandomized
study, 138 medical students performed significantly better on clinically related multiple-choice
questions, p < 0.05, relative to fellow dental students who participated in traditional lectures
alone.[10] It is also important when assessing the effect of a teaching intervention to consider
the proximity of the intervention relative to the time of testing.[11] Finally, participant preference should be considered. For example, 44 ophthalmology
residents were randomized into two groups. One group took the flipped glaucoma classroom
and lecture-based ocular trauma classroom, while the other group took the flipped
ocular trauma classroom and lecture-based glaucoma classroom. There was no difference
in the posttest examination between groups and the residents and attendings rated
the flipped classroom as more desirable. Eighty percent of the group suggested one
flipped classroom per week as the optimal exposure to the flipped classroom venue.[2]
For additional consideration, there are advantages to synchronous and asynchronous
learning and both lend themselves to traditional lectures. The Accreditation Council
for Graduate Medical Education review committee for ophthalmology allows a great range
in learning option to satisfy the 360-hour requirement over the course of a residency.[12] Emergency medicine has made use of asynchronous learning, as it is difficult to
have everyone present at the same location at the same time. It allows on-demand viewing
at one's own pace, stopping, starting, and reviewing as desired. It can be as simple
as recording a lecture for later use, an adjunct to a flipped classroom, a prelude
to problem-based learning, or it can allow review of periodic regulatory material
that would otherwise fill valuable learning time. In a prospective randomized study
comparing emergency medicine residents receiving synchronous versus asynchronous instruction,
the synchronous group scored better initially but at the 10-week retest there was
no difference in performance. The two formats had similar degrees of acceptability.[13]
The largest shortcomings of the study presented here were lack of randomization and
the small sample size. Without randomization lecture hour attendance cannot be separated
from potentially confounding variables such as interest in the subject, competing
events, and other factors that might link lecture attendance with OKAP performance.
Additionally, the small sample size was accentuated when subdividing further in the
PGY group analysis. The sample size precluded looking at important factors such as
lecturer, timing relative to OKAPS, Basic and Clinical Science Course section, innate
individual test-taking skills, and call and rotation duties prior to the examinations.
Finally, with the small size and single program makeup of this study, generalization
to all ophthalmology programs or other specialties warrants caution.
Other factors worth investigating, that may prove useful in predicting the attainment
of ophthalmic knowledge, might include determination of baseline ophthalmic understanding
and test-taking skills and tracking what tools are used to study and the duration
each is used. The many inputs from the learning tools combined with common assessment
tools such as OKAPs, written board scores, and oral board pass rates complement artificial
intelligence platforms such as the Kalman filtering process. With standardization
of assessing surgical skill performance or any of the milestone it is possible, by
pooling data from many programs, to project resident performances, detect problem
areas sooner, and individualize learning plans. For example, pooled interprogram data
might suggest that 30 hours (95% CI 20–35) of simulation training is required for
capsulorhexis proficiency. This would be useful information before a resident embarks
on a rotation as the primary surgeon in a high volume general ophthalmology rotation.
This pooling of data has been successfully done with assessment of glaucoma progression[14]
[15] as well as many unrelated disciplines with the commonality of multiple inputs, measurable
outcomes, and many data points. Finally, providing the residents with statistically
based feedback from learning studies can assist in changing behavior.
