Simulation-based training improves polypectomy skills among practicing endoscopists

Abstract

Background and study aims Practicing endoscopists have variable polypectomy skills during colonoscopy and limited training opportunities for improvement. Simulation-based training enhances procedural skill, but its impact on polypectomy is unclear. We developed a simulation-based polypectomy intervention to improve polypectomy competency.

Methods All faculty endoscopists at our tertiary care center who perform colonoscopy with polypectomy were recruited for a simulation-based intervention assessing sessile and stalked polypectomy. Endoscopists removed five polyps in a simulation environment at pretest followed by a training intervention including a video, practice, and one-on-one feedback. Within 1–4 weeks, endoscopists removed five new simulated polyps at post-test. We used the Direct Observation of Polypectomy Skills (DOPyS) checklist for assessment, evaluating individual polypectomy skills, and global competency (scale: 1–4). Competency was defined as an average global competency score of ≥ 3.

Results 83 % (29/35) of eligible endoscopists participated and 95 % (276/290) of planned polypectomies were completed. Only 17 % (5/29) of endoscopists had average global competency scores that were competent at pretest compared with 52 % (15/29) at post-test (P = 0.01). Of all completed polypectomies, the competent polypectomy rate significantly improved from pretest to post-test (55 % vs. 71 %; P < 0.01). This improvement was significant for sessile polypectomy (37 % vs. 65 %; P < 0.01) but not for stalked polypectomy (82 % vs. 80 %; P = 0.70).

Conclusions Simulation-based training improved polypectomy skills among practicing endoscopists. Further studies are needed to assess the translation of simulation-based education to clinical practice.

Introduction

Colonoscopy is the cornerstone for polyp detection and resection during colorectal cancer screening. Current quality indicators focus predominantly on polyp detection (e. g. adenoma detection rate [ADR]) over adequacy of resection. Although achieving ADR thresholds predicts a reduction of interval cancers [1], recent data suggest limited correlation between ADR and adequacy of polypectomy [2]. Research also shows that incomplete polypectomy can lead to interval colon cancers [3]. Furthermore, endoscopists perform polypectomy with significant variability in skill. Recent research shows the rate of competent polypectomy ranging from 30 % to 90 % among practicing endoscopists [2].

Although recent literature has suggested the implementation of hands-on curricula to improve polypectomy skill [4] [5], the effectiveness of such training among practicing physicians has not been established. Endoscopy skills training occurs during fellowship usually via the teacher–apprentice model, where trainees learn vicariously from experienced endoscopists. By contrast, simulation is an effective learning technology that has been used in gastroenterology fellowship and with novice endoscopists to supplement traditional training [6] [7] [8] [9]. However, simulation is not used as part of continuing medical education (CME) beyond focused endoscopy courses. Endoscopy CME courses also lack a structured approach to skill assessment. There is limited opportunity for well-defined CME polypectomy training and assessment with feedback among practicing endoscopists.

The Direct Observation of Polypectomy Skills (DOPyS) checklist is an assessment measure with validity evidence supporting its utility for polypectomy competency assessment [10] [11]. The DOPyS tool has been used in several studies to evaluate polypectomy competency in a set of specific and global clinical skills [2] [4] [12]. Prior studies have used its global competency measure to compare clinical fitness at fixed intervals while using the individual checklist items to provide structured feedback [4] [12].

Given the variability of polypectomy skills, the lack of defined training opportunities, and the potential of simulation to boost skills, we developed and evaluated a simulation-based polypectomy curriculum specifically for practicing endoscopists. We assessed polypectomy competency before and after simulation-based training utilizing the DOPyS checklist. We hypothesized that implementation of a simulation-based intervention for practicing endoscopists would improve their polypectomy skills.

Methods

Study design and participants

We performed a prospective pretest–post-test study at Northwestern Memorial Hospital, from July to August 2018, to assess practicing endoscopists’ simulated polypectomy skills. All faculty at our center who perform colonoscopy with polypectomy were eligible for the study and were invited to participate. The study was deemed exempt by the Northwestern University Institutional Review Board (STU00207632).

Polypectomy simulation model

An ex vivo bovine model was created for the simulation experience (EndoSim, LLC, Hudson, Massachusetts, USA). Each model was designed to present 10 polyps (6 sessile and 4 stalked lesions) in a standardized fashion. Sessile lesions were marked with red-dye and outlined with electrocautery. Stalked lesions were created through implantation of tissue under the mucosal surface ([Fig. 1]). Sessile and stalked lesions were designed to range in size from 8 mm to 15 mm. The model was arranged in a sigmoid “S” shape and secured to simulate rectal insertion ([Fig. 2]). The model tissue allowed for all colonoscopy functions to be used including water irrigation, insufflation, and polypectomy. Simulation-based training and testing were initially conducted with nine gastroenterology fellows (postgraduate years 4–6). Feasibility and technical aspects of the model to perform competency skills were assessed and adjustments were made based on feedback to improve the reliability of the model.

Fig. 1 Simulated stalked polyp removal.

Fig. 2 Simulation model configuration.

Procedure

The simulation environment was designed to replicate an endoscopy procedure room with a complete colonoscopy tower, high definition colonoscope, standard polypectomy tools, electrocautery, and an experienced technician or registered nurse ([Fig. 3]). Participants first provided demographic data including age, sex, years in practice and estimates of the average number of colonoscopies and polypectomies ( > 5 mm) performed annually.

Fig. 3 Set up of simulated polypectomy experience.

During the first simulation session, participants were paired with a trainer (R.N.K., S.K., A.R., S.B.W.). Participants took a pretest on the simulator where they were instructed to remove a total of five polyps (three sessile, two stalked) observed by the trainer. Each participant then viewed a brief video on polypectomy best practices developed by an interventional endoscopist (S.K.). Participants next underwent one-on-one proctored training with an experienced endoscopist (R.N.K., S.K., A.R., S.B.W.) for 45 minutes to practice polypectomy skills on the simulator, and feedback was provided. The remaining five polyps within the model were used for the proctored training. During training, the proctors emphasized competent polypectomy maneuvers found in the DOPyS checklist. Feedback focused on areas for improvement based on pretest and training observations.

Endoscopists returned for the second simulation session within 1–4 weeks. This session involved a simulated post-test, during which endoscopists removed five new polyps (three sessile, two stalked) ([Fig. 4]).

Fig. 4 Simulation protocol and study design.

We video recorded the endoscopic view of all testing sessions. Video recordings were obtained and edited using a video cloud storage service (Virgo Surgical Video Solutions, Inc., San Francisco, California, USA). All videos were edited to include only the polypectomy, from polyp identification to polyp retrieval. Each video received a de-identified study number. Four experienced endoscopist assessors reviewed all videos (R.N.K., S.K., A.R., S.B.W.). Two of the four assessors evaluated each pretest and post-test polypectomy video recording using the DOPyS checklist [10] [11]. Reviewers were blinded to the endoscopist and session (pre vs. post-test).

Assessment measure

The DOPyS checklist, used for simulated polypectomy assessment, is designed to evaluate the technical skills and competency of polypectomy [10] [11]. The 33 checklist items are structured to evaluate a) preprocedural or generic skills, b) polypectomy skills specific to sessile or stalked lesions, and c) post-polypectomy skills. The DOPyS checklist also provides a global competency score for assessment of overall competency. Each checklist item and the global competency score are scored from 1 to 4, with scores of 1 and 2 corresponding to suboptimal performance, and scores of 3 and 4 corresponding to competent performance. The DOPyS measure yields reliable data when at least two assessors evaluate at least five polyps removed by the same endoscopist [10]. The DOPyS checklist permits evaluation in both live and video-recorded settings [10] [11].

The DOPyS measure was modified for final video-based assessment of simulated polypectomy. Five of the six items that require live assessment were excluded (checks all polypectomy equipment, checks snare closure, clear instructions to staff, checks diathermy settings, checks for retrieval of polyp). Items that could not be replicated in a simulated model were also excluded (i. e. items related to hemostasis, tattoo usage). The final modified DOPyS checklist used in our study contained 23 items: 9 items pertained to all polyps (6 preprocedural or generic skills, 3 post-polypectomy skills) and 14 were specific to polyp type (7 stalked polypectomy skills, 7 sessile polypectomy skills). Thus, for each stalked or sessile polyp, 16 items were rated in addition to a global competency rating.

Assessor training

Assessors attended a “train the trainer” session before the first candidate evaluation session. The DOPyS checklist was reviewed and discussed in detail. Assessors were educated about each aspect of scoring and received descriptors for each checklist item.

Competency assessment

For each polyp, individual checklist items and global competency were scored by two assessors. In the event that one assessor scored the global competency as suboptimal and the second assessor scored it as competent, adjudication was performed by a third assessor. We then assessed competency per participant by averaging global competency scores for polyps removed at pretest and post-test. We also examined polyp competency rates for all polyps, as well as competency for individual checklist items at pretest and post-test. Competency was defined as an averaged score of ≥ 3.

Study outcomes

Our primary end point was average global competency for all polyps removed. Our secondary end points included polypectomy competency per polyp type and per individual DOPyS checklist item at pre- and post-test.

Statistical analysis

Participant characteristics were reported as percentages and means with standard deviations (SDs) or medians with interquartile ranges for continuous variables depending on data distribution. We assessed interobserver checklist agreement on pre- and post-tests using Cohen’s Kappa coefficient. We used the Fisher’s exact test to compare participant’s average global competency achievement from pretest to post-test. Chi-squared analysis was used to compare rates of competent polypectomy at pretest and post-test for all polyps. We used paired t tests to compare global competency scores of all polyps and individual checklist items from pre- to post-test.

Statistical analyses were performed using IBM SPSS version 25 (IBM Corp., Armonk, New York, USA).

Results

Participants and polypectomy characteristics

At the time of the study, our center had 35 faculty endoscopists who performed colonoscopy with polypectomy, 29 of whom (83 %) participated in the study. The majority of endoscopists were male (66 % [19/29]) and the mean duration of practice experience was 15.07 (SD 11.31) years. Endoscopists performed a median of 500 colonoscopies and 100 polypectomies (> 5 mm) per year ([Table 1]). Out of 290 planned polypectomies (29 endoscopists each with five pretest and five post-test polypectomies), 276 polypectomies were used for final analysis (141 pre, 135 post). Four pretest and four post-test polypectomies were not included due to equipment failure, and six post-test polypectomies were not included due to the participant terminating the session early (one participant elected to not finish the post-test entirely, one participant did not remove all five requested polyps).

Table 1 Baseline demographics of endoscopist participants.

Characteristic	Endoscopists (n = 29)
Female, n (%)	10 (34)
Age, mean (SD), years	47.43 (10.83)
Years in practice, mean (SD)	15.07 (11.31)
Colonoscopies per year, median (IQR)	500 (100, 1000)
Polypectomies (> 5 mm) per year, median (IQR)	100 (30, 300)
Endoscopists who reported feeling comfortable performing polypectomy, n (%)	27 (93)

SD, standard deviation; IQR, interquartile range.

Polypectomy competency

The interobserver reliability of global competency among assessors showed moderate agreement (κ = 0.67). In 14 % (39/276) of polypectomies, there was a discrepancy in global competency scores between assessors, which required additional review by a third assessor.

Overall, 17 % (5/29) of the endoscopists had average global competency scores that were competent at pretest compared with 52 % (15/29) at post-test (P = 0.01). At pretest, 55 % (78/141) of polyps were removed competently compared with 71 % (96/135) at post-test (P < 0.01). Global competency scores significantly improved from pretest to post-test for all polyps (2.62 [SD 0.79] vs. 2.86 [SD 0.84]; P = 0.02).

For sessile polypectomy, 37 % (31/84) of polyps were removed competently at pretest compared with 65 % (53/81) at post-test (P < 0.01). For stalked polypectomy, 82 % (47/57) of polyps were removed competently at pretest compared with 80 % (43/54) at post-test (P = 0.70). There was a significant improvement from pretest to post-test for sessile polyps (global competency score: 2.36 [SD 0.79] vs. 2.73 [SD 0.88]; P = 0.005) but not for stalked polyps (global competency score: 3.01 [SD 0.61] vs. 3.06 [SD 0.77]; P = 0.72).

Individual polypectomy skill competency

Photo-documentation significantly improved from pretest to post-test (1.56 [SD 0.67] vs. 1.97 [SD 0.98]; P < 0.001). Four individual DOPyS checklist item scores significantly improved for sessile polyp sub-components: proceeding only if the lesion lifts adequately, selecting the appropriate snare size, appropriate position of snare over the lesion, and ensuring the appropriate amount of tissue is trapped within the snare ([Table 2]). No individual skills for stalked polypectomy improved from pretest to post-test, though baseline performance was already high ([Table 2]). Post-polypectomy, identifying and treating residual polyp tissue significantly increased from pretest to post-test (2.58 [SD 0.92] vs. 2.86 [SD 0.96]; P = 0.01).

Table 2 Individual Direct Observation of Polypectomy Skills scores pre and post simulation-based intervention.

Item	Pre-test (n = 141 assessments)	Post-test (n = 135 assessments)	P value
Pre-polypectomy
Optimizes polyp position	2.98 (0.54)	3.04 (0.69)	0.46
Optimizes polyp view	3.00 (0.59)	3.07 (0.69)	0.36
Determines full extent of lesion	3.05 (0.55)	3.11 (0.67)	0.41
Adjusts/stabilizes colonoscope position	3.04 (0.50)	3.11 (0.64)	0.30
Uses appropriate polypectomy technique	3.34 (0.45)	3.44 (0.59)	0.10
Photo-documents pre- and post-polypectomy	1.56 (0.67)	1.97 (0.98)	< 0.001
Stalked polyps	n = 57	n = 54
Selects appropriate snare size	3.35 (0.38)	3.43 (0.63)	0.45
Directs snare accurately over polyp head	3.25 (0.48)	3.14 (0.72)	0.36
Correctly selects en bloc or piecemeal removal depending on size	3.39 (0.41)	3.30 (0.71)	0.42
Advances snare sheath toward stalk as snare closed	3.26 (0.51)	3.14 (0.69)	0.28
Places snare at appropriate position on the stalk	3.20 (0.53)	3.06 (0.78)	0.28
Mobilizes polyp to ensure appropriate amount of tissue is trapped within snare	3.19 (0.51)	3.06 (0.67)	0.26
Applies appropriate degree of diathermy	3.34 (0.40)	3.34 (0.63)	0.99
Sessile polyps	n = 84	n = 81
Adequate submucosal injection	2.68 (0.70)	2.83 (0.81)	0.19
Proceeds only if the lesion lifts adequately	2.82 (0.71)	3.12 (0.72)	0.009
Selects appropriate snare size and directs snare accurately over the lesion head	2.52 (0.72)	2.78 (0.86)	0.04
Correctly selects en bloc or piecemeal removal depending on size	2.64 (0.76)	2.85 (0.94)	0.12
Appropriate position of snare over lesion as snare closed	2.44 (0.77)	2.71 (0.92)	0.04
Ensures appropriate amount of tissue is trapped within snare and tents lesion gently away from the mucosa	2.39 (0.76)	2.70 (0.93)	0.02
Uses cold snare technique or applies appropriate diathermy as applicable	2.93 (0.63)	3.07 (0.90)	0.27
Post-polypectomy	n = 141	n = 135
Examines remnant stalk/polyp base	2.91 (0.76)	2.93 (0.90)	0.86
Identifies and appropriately treats residual polyp	2.58 (0.92)	2.86 (0.96)	0.01
Retrieves or attempts retrieval of polyp	2.8 (0.83)	2.85 (1.02)	0.68

All values are reported as mean (SD).

Discussion

We developed a simulation-based intervention for polypectomy training and assessed its utility among practicing endoscopists. Simulation-based training significantly improved polypectomy skills in our cohort. As the majority of colonoscopy and polypectomy education targets trainees in the United States, formal endoscopy training for practicing gastroenterologists is limited to professional society courses and industry-driven proctorships. Prior studies have highlighted the high degree of variation in endoscopists’ polyp resection efficacy and technical competency [2] [13]. Implementation of a well-designed training program for practicing endoscopists adds value by identifying specific deficits and improving polypectomy skills.

Our study not only highlights the utility of simulation-based education, but also brings to light the need for polypectomy improvement. The current study was designed to include polyps 8–15 mm in size, and assessed sessile and stalked polypectomy. This size range was chosen to assess skills for small-to-large lesions, specifically stalked polyps, which would require snare cautery, and sessile polyps amenable to submucosal injection followed by resection. Prior work has described suboptimal polypectomy in small-to-large polyps (≥ 6 mm), as well as low rates of submucosal injection, a common skill required for sessile polypectomy [2]. In this study, submucosal injection was often discussed and emphasized during hands-on feedback. Competency for sessile polypectomy improved significantly, whereas stalked polypectomy did not. Based on prior literature, subjects are more likely to be assessed as competent for stalked polypectomy at baseline and benefit most from training directed at removal of flat polyps requiring submucosal injection. Although the overall effect of our intervention was modest, the improvement in sessile polypectomy was likely the major contributor toward increased competency at post-test.

Research studies estimate that nearly 20 % of interval cancers are related to incomplete polyp resection [3]. In earlier work examining adequacy of resection, 10 % of margins in 346 resected polyp sites had residual adenomatous tissue [13]. In a recent similarly conducted study, Pedersen et al. identified that 16 % of polyps were incompletely resected [14]. The incomplete resection rate was higher for larger and sessile lesions [13]. This study shows an improvement in assessing residual polyp tissue from pretest to post-test. While the overall competency improvement seen from pretest to post-test reflects the heterogeneous effect of individual skills, identifying and appropriately treating residual polyp tissue in a simulated setting may have downstream translation to patient care.

Development of simulation curricula and checklists for procedural skills allow for advancement of competency assessment. However, the threshold of skill an endoscopist should meet or exceed (i. e. minimum passing standard [MPS]) specifically for polypectomy has not been established. Additionally, traditional pretest–post-test interventions do not allow for participants to cycle back through the training until skill mastery is achieved. Our goal in the current study was to implement simulation-based training in a large practice of practicing endoscopists and to demonstrate the use of simulation to improve clinical skill. In future studies, we plan to implement simulation-based mastery learning, a rigorous form of competency-based education that defines an MPS. In future CME endoscopy training, all participants will be expected to complete deliberate practice until the MPS is achieved. Therefore, results will be uniform while education time may vary [15]. In an era where competency-based education is prevalent for trainees [16], continuing education for our faculty should meet or exceed this expectation in a psychologically safe culture that allows for feedback, skill enhancement, and achievement of specific outcomes.

This study has several limitations. Simulated environments do not fully represent the real-life patient–physician experience. To address this, extensive steps were taken to simulate a real-world experience. Ex vivo models also have limits. Many iterations were performed using trainee trials to create an effective model. Polyp creation and presentation was standardized regularly. There may also be limitations of using the DOPyS checklist in ex vivo models. Despite such constraints, the assessments made using the modified DOPyS checklist are comprehensive for preprocedural, intraprocedural, and post-procedural assessment. The interobserver reliability among assessors showed moderate agreement. Further studies may need more checklist training for assessors. Not all endoscopists who were eligible participated in the study; though the number was small, further efforts are required to enroll all participants and to avoid evaluation apprehension. Another condition is that simulation skills improvement may be related to improved pre- to post-test familiarity with the model. However, many checklist items have both cognitive and technical elements, and therefore familiarity with the model may not be influential. Finally, translation to clinical practice cannot be determined from this study. Studies assessing faculty competency in video-recorded patient polypectomies are under way to study translation of simulation-based training to clinical practice.

In conclusion, we present a simulation-based intervention that significantly improved polypectomy skills among practicing endoscopists. The competency improvement, particularly with sessile polypectomy, highlights the ability of simulation to advance procedural skills among practicing endoscopists. More long-term assessment and training are needed to ensure all endoscopists reach rigorous mastery standards and to demonstrate how simulation-based polypectomy training translates to clinical practice.

Competing interests

Dr. Wani is a consultant for Medtronic, Boston Scientific, Interpace, and Cernostics. He is supported by the University of Colorado Department of Medicine Outstanding Early Scholars Program. The remaining authors declare that they have no conflict of interest.

• References

• 1 Kaminski MF, Regula J, Kraszewska E. et al. Quality indicators for colonoscopy and the risk of interval cancer. N Engl J Med 2010; 362: 1795-1803
• 2 Duloy AM, Kaltenbach TR, Keswani RN. Assessing colon polypectomy competency and its association with established quality metrics. Gastrointest Endosc 2018; 87: 635-644
• 3 Robertson DJ, Lieberman DA, Winawer SJ. et al. Colorectal cancers soon after colonoscopy: a pooled multicohort analysis. Gut 2014; 63: 949-956
• 4 Duloy AM, Kaltenbach TR, Wood M. et al. Colon polypectomy report card improves polypectomy competency: results of a prospective quality improvement study (with video). Gastrointest Endosc 2019; 89: 1212-1221
• 5 van Doorn SC, Bastiaansen BA, Thomas-Gibson S. et al. Polypectomy skills of gastroenterology fellows: can we improve them?. Endosc Int Open 2016; 4: E182-E189
• 6 Walsh CM, Sherlock ME, Ling SC. et al. Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. Cochrane Database Syst Rev 2012; 06: CD008237
• 7 Singh S, Sedlack RE, Cook DA. Effects of simulation-based training in gastrointestinal endoscopy: a systematic review and meta-analysis. Clin Gastroenterol Hepatol 2014; 12: 1611-1623
• 8 Grover SC, Garg A, Scaffidi MA. et al. Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial. Gastrointest Endosc 2015; 82: 1072-1079
• 9 McConnell RA, Kim S, Ahmad NA. et al. Poor discriminatory function for endoscopic skills on a computer-based simulator. Gastrointest Endosc 2012; 76: 993-1002
• 10 Gupta S, Anderson J, Bhandari P. et al. Development and validation of a novel method for assessing competency in polypectomy: direct observation of polypectomy skills. Gastrointest Endosc 2011; 73: 1232-1239
• 11 Gupta S, Bassett P, Man R. et al. Validation of a novel method for assessing competency in polypectomy. Gastrointest Endosc 2012; 75: 568-575
• 12 Ansell J, Hurley JJ, Horwood J. et al. Can endoscopists accurately self-assess performance during simulated colonoscopic polypectomy? A prospective, cross-sectional study. . Am J Surg 2014; 207: 32-38
• 13 Pohl H, Srivastava A, Bensen SP. et al. Incomplete polyp resection during colonoscopy – results of the complete adenoma resection (CARE) study. Gastroenterology 2013; 144: 74-80
• 14 Pedersen IB, Bretthauer M, Kalager M. et al. Incomplete endoscopic resection of colorectal polyps: a prospective quality assurance study. Endoscopy 2021; 53: 383-391
• 15 McGaghie WC. Mastery learning: it is time for medical education to join the 21st century. Acad Med 2015; 90: 1438-1441
• 16 Gruppen LD. Competency-based education, feedback, and humility. Gastroenterology 2015; 148: 4-7

Publication History

Received: 13 January 2021

Accepted: 04 June 2021

Article published online:
12 November 2021