The great expansion of Medical Residency Programs in Brazil in recent decades has
made it difficult to properly evaluate the trained professionals and the quality of
training offered.
Today, ∼53,776 physicians are registered at the National Medical Residency Commission
enrolled in 4,862 Medical Residency Programs offered by 809 institutions. Only in
Gynecology and Obstetrics (Ob-gyn), there are 312 Medical Residency Programs. Despite
the recommendation that resident physicians undergo quarterly theoretical and practical
evaluations by the programs, this has not been happening regularly. Therefore, there
is no information about the performance of residents during their training and the
quality of training programs. Evidently, this knowledge should be based on information
obtained during ongoing evaluations of the programs, with visits, audits and reports,
although this has not happened in practice.
According to current legislation, specialists graduated from Medical Residency Programs
approved by the National Medical Residency Commission automatically receive the specialist
certificate recognized by the Ministry of Education and the Federal Council of Medicine
without any evaluation process of the students' competences at the end of the program.
In view of the fragility of the system, the Medical Specialties Societies in Brazil,
with special authorization from the Brazilian Medical Association, started to grant
the title of specialist for graduates of residency programs after an evaluation through
a flexible exam, depending on the Society granting it. In this sense, the Title of
Specialist in Gynecology and Obstetrics (Portuguese acronym: TEGO) given by the Brazilian
Federation of Gynecology and Obstetrics Associations (FEBRASGO) is only granted upon
evaluation of the candidate through theoretical and practical tests carefully prepared
by the National TEGO Commission of FEBRASGO.
In this evaluation process, important failures in the training of new specialists
who completed the residency program have been observed, which has raised the failure
rates for the TEGO. Thus, considering the need to qualify the training of Ob-gyn specialists
in Brazil and understanding that the Individual Progress Test (IPT) represents a great
reference for the self-assessment and improvement of resident physicians and Medical
Residency Programs, as of 2018, FEBRASGO has implemented the Individual Progress Test
for Ob-Gyn Residents (IPT-GO). Although the IPT is widely used internationally, in
Brazil there is little experience reported on its use in Medical Residency.
The IPT is a comprehensive assessment that preferably uses multiple choice questions
and is periodically applied to all students of the same curriculum or program,[1] aligned with modern constructivist education and promotes long-term knowledge. The
longitudinality of this modality of assessment provides a unique and demonstrable
measure of students cognitive progression.[2]
The functional purpose of the IPT is to provide reliable information for self-assessment
of candidates and service providers of Medical Residency Programs hence, it is a formative
assessment. For resident physicians who take the test, the IPT provides an accurate
measure of their level of knowledge in relation to their peers and in relation to
the final objectives of the specialty training, according to the Gynecology and Obstetrics
Competence Matrix.[2]
[3]
Furthermore, through the performance in serial evaluations, the individual progress
of the cognitive component can be evaluated. After each assessment, it is possible
to reaffirm and consolidate knowledge and identify learning gaps and points to be
improved.[4]
For preceptors and supervisors of Medical Residency Programs, the IPT-GO provides
information on the performance and progression of residents who have taken the exam.
Through this information, the profile of residents admitted to each service, the added
knowledge of residents throughout the training program and the level of knowledge
of graduates in relation to the national average and in relation to the objectives
of the Competence Matrix can be assessed. It also allows the identification of strengths
and points of improvement or areas requiring reinforcement for learning.[4]
Since 2018, the IPT began to be offered annually to all resident physicians in the
first (R1), second (R2) and third years (R3) of training regularly enrolled in Medical
Residency Programs recognized by the Ministry of Education.
In 2018 and 2019, the test was applied in person and simultaneously in 11 Brazilian
cities in regions with the highest concentration of Ob-Gyn Residency Programs. In
2020, due to the Covid-19 pandemic and in compliance with sanitary requirements, the
IPT could not be applied in person. Faced with this new challenge, the choice was
to apply the test online, since this measure is supported by the literature.[5]
The theoretical test for obtaining the TEGO, applied annually to newly graduated specialists
in the field, is a comprehensive assessment of the skills provided for in the Gynecology
and Obstetrics Competence Matrix[3] and used as a reference to the final level of Residency Programs. For these reasons,
this test model was chosen to be applied in the IPT-GO.[6]
Criteria ranging from a bonus of points to exemption from the TEGO theoretical test
in the year following the completion of the residency program were established as
a form of encouraging residents' participation in the IPT-GO. These criteria are based
on adherence and individual performance on the test. To be entitled to bonuses, it
is an essential condition that the resident participates in all three versions of
the IPT as R1, R2 and R3, with a progressive minimum performance established in accordance
with the competition notice published each year.[7]
Individual Feedback to Candidates
Feedback on the results of each candidate's performance in the IPT-GO is provided
confidentially through a password-protected online system with personal access. The
aim of this measure is to avoid embarrassment, discrimination or disqualification
of candidates with unsatisfactory performance. Therefore, this evaluation is not intended
to rank candidates or services. The online system presents graphs where candidates
can assess the progression of their individual performance and compare it to their
peers through the median (Me), 30th percentile (P30) and 60th percentile (P60) of the overall performance. For candidates completing residency
programs (R3), in addition to the evaluation of the performance level in the triennium,
information on whether or not they have received a bonus for TEGO of the following
year is also provided.
Feedback to Services and Residency Programs
For Medical Residency Programs, feedback is provided by FEBRASGO directly to preceptors
or responsible persons. The information provided corresponds to the median performance
of each category of residents (R1, R2 and R3) of the service compared with the Median,
P30 and P60 values of the overall performance without identifying the residents.
Analysis of Results
In 2020, the IPT-GO completed its third edition, making it possible to assess the
performance of the first complete cohort of residents who took the test in the three
consecutive years of Ob-Gyn Residency Programs in Brazil ([Table 1]).
Table 1
Number of participants in the three Residency levels over the three years of application
of the Individual Progress Test
|
Level
|
2018
n (%)
|
2019
n (%)
|
2020
n (%)
|
|
R1
|
497(41)
|
568(43)
|
628(44)
|
|
R2
|
360(30)
|
457(35)
|
480(34)
|
|
R3
|
345(29)
|
289(22)
|
314(22)
|
|
Total
|
1202
|
1314
|
1422
|
|
Increment
|
|
9.3
|
7.6
|
Among candidates, female participation was 85.6%, 85.9% and 85.5% respectively in
2018, 2019 and 2020, in a clear demonstration that Ob-Gyn have become a predominantly
women's specialty. The total number has increased over the three years, particularly
due to R1 increment, since a decrease in R2 and R3 was observed. Some factors should
be mentioned in this “balance” of gains and losses in these numbers. Undoubtedly,
the bonus and exemption opportunities in the TEGO theoretical test have contributed
to encourage longitudinal participation of residents. However, a large part of the
discontinuity may be attributed to the poor performance of some candidates in the
first versions of the test, resulting in dropouts due to the impossibility of bonus
or exemption from the TEGO theoretical test. Another noteworthy factor that justifies
non-adherence a considerable part of the residents would be the difficulty of program
coordinators in releasing candidates on the day of the exam, considering the resident's
need for work given the composition of the teams on duty at the respective hospital.
As demonstrated, in all editions of the IPT-GO there was a progressive improvement
in performance among candidates from the first (R1), second (R2) and third (R3) years
of the Ob-Gyn Residency Program. This reflects the knowledge increase occurring throughout
the training period in most programs. It also reflects the quality of the test, which
consists of a comprehensive, valid and reliable assessment of all knowledge expected
by those completing the program (R3) ([Table 2]).
Table 2
Median (Me), 30th (P30) and 60th (P60) percentiles of the grades assigned to residents at different levels who took
the Individual Progress Test-GO in 2018, 2019 and 2020
|
Progress Test - Performance of Gynecology and Obstetrics residents by level of treatment
|
|
2018
|
2019
|
2020
|
|
R1
|
R2
|
R3
|
R1
|
R2
|
R3
|
R1
|
R2
|
R3
|
|
P30
|
4.8
|
5.2
|
5.5
|
5.5
|
5.8
|
6.1
|
5.1
|
5.4
|
5.8
|
|
Me
|
5.2
|
5.6
|
5.9
|
5.9
|
6.2
|
6.6
|
5.5
|
5.8
|
6.3
|
|
P60
|
5.5
|
5.8
|
6.1
|
6.1
|
6.4
|
6.8
|
5.6
|
6.0
|
6.5
|
Performance of the Cohort of Residents from 2018 to 2020
The performance evaluation of 314 resident physicians who started the IPT-GO in 2018
as R1 and completed the last version in 2020 deserves attention. This cohort of candidates
represents the first group that underwent the three versions of the IPT-GO since its
implementation, and serves as a reference for various analyzes and considerations.
The median grades of performance of this group in 2018, 2019 and 2020 were respectively
5.2(R1) 6.2(R2) and 6.3(R3). A progression in the performance of the same group of
residents from the beginning to the end of their training period can be observed,
reflecting the acquisition of knowledge during the residency program. These results
are consistent with a well-structured IPT, which is comprehensive and focused on the
content expected for graduates of the Program.[2]
The smaller increase in performance between years 2019 and 2020 may somehow be a reflection
of the Covid-19 pandemic period that certainly affected the performance of all residents
indistinctly, given the huge restrictions imposed on in-person activities, particularly
in outpatient and surgical practices.[8]
In conclusion, the implementation of the IPT for Ob-Gyn residents across the country
was a great experience in several aspects.
The wide dissemination of the IPT on social networks, on FEBRASGO's institutional
Web site, at scientific events in the specialty and through direct mail to all Ob-Gyn
residents and Medical Residency Programs in the country allowed FEBRASGO to get closer
to residents, improve and update the registration of Medical Residency Programs, preceptors
and other relevant information. The number of residents associated with FEBRASGO has
increased considerably in the last two years.
The form of performance feedback to program coordinators allowed for the comparison
of results with similar services throughout the country and motivated reflections
and internal debates among coordinators, preceptors and residents, and drew attention
to the need for improvement in many programs. Note that institutional commitment through
the support of program directors, coordinators and preceptors is key, as there is
a need to release residents from the shift schedule on the day of the test.
Information from the Individual Progress Test can be used consistently for diagnostic,
prognostic and corrective learning through self-assessment and structured feedback.
When compared with final summative evaluations, the IPT provides greater support and
security in making high-impact decisions such as approval, failure and progression
of the student. Additionally, longitudinal data can serve as a measure of the quality
and transparency of programs and achievement of curriculum objectives by educational
institutions.
Considering the immense difficulties encountered by government agencies to audit Medical
Residency Programs on a permanent basis, this model may be adopted and validated by
the National Medical Residency Commission as one of the parameters for evaluating
Medical Residency Programs through partnerships with the Medical Specialties Societies.
It is worth mentioning that all expenses arising from the application of the test
were made with FEBRASGO's own resources so to not burden the National Medical Residency
Commission.
