Hospital compound-level endoscopy training quality performance: scoping the spectrum

Introduction

Endoscopy training is a multidisciplinary activity, the quality of which is assured by the Joint Advisory Group on Gastrointestinal Endoscopy (JAG). Cross-specialty training is registered via the Joint Advisory Group on Gastrointestinal Endoscopy Training System (JETS), with certification awarded on completion of appropriate courses, direct observation of procedural skills (DOPS) assessments [1] [2] [3] , critical indicative procedures, and key performance indicators (KPIs).

JAG recommends weekly, adjusted training lists of eight points; with esophagogastroscopy and flexible sigmoidoscopy allocated one point, and colonoscopy two points, equating to 20 annual training lists. Currently the quality assurance of endoscopy training is assessed by the Global Rating Scale (GRS) in all UK endoscopy units. This is a biannual, self-assessed score rating three training domains (training environment, trainers, and assessment and appraisal), included in the JAG accreditation standards, assessed on JAG visits. Yet does this provide a sufficiently detailed and objective measure of the delivery of training at the unit level?

A UK 2015 training survey reported a disparity in access between medical gastroenterologists and surgeons (0.9 vs. 0.5 lists per week, and 3.0 vs. 1.2 ad hoc training opportunities) [4] and following the Department of Health’s initiative to improve service, the Clinical Endoscopy Training Programme was piloted [5] with a standard of two weekly sessions to promote faster certification.

In the surgical arena, accommodating endoscopy training within operative curricula and educational contracts, aimed at dual accreditation, is especially challenging. Moreover, the COVID-19 pandemic has resulted in sweeping cutbacks in endoscopy services: overall activity has fallen to 12 % of pre-COVID level, cancer detection by 58 %, and training by 93 % (mean 1930 to 133 procedures per week), performed by only 46 trainees [6]. The aim of this study was to assess the quality of endoscopy training in a single UK Statutory Educational Body (SEB) related to hospital units compared with JETS certification standards.

Methods

Data recorded by 211 consecutive cross-specialty trainee endoscopists registered with the JAG on Gastrointestinal Endoscopy Training System (JETS) across 18 hospital endoscopy units throughout 2019, amounting to 28,928 training procedures, were analyzed. Data included the numbers of trainers and trainees, frequency of training lists, procedures performed, Direct Observation of Procedural Skills (DOPS) completed, and Key Performance Indicators (KPI). To satisfy JAG certification guidelines, 20 completed DOPS should be obtained over the course of 200 procedures; DOPS per 200 procedures related to unit were estimated by dividing the total procedures performed per unit by the number 200. ERCP and endoluminal ultrasound were excluded from the analysis because of their status as advanced specialist endoscopic procedures, with fewer hospitals equipped to deliver such training across Wales. The JAG provided ethical approval.

Statistical analysis

One sample Wilcoxon-signed rank test was used to compare observed metrics against JAG standards. Descriptive statistics were used to describe differences in training activity across individual hospital units where no comparative standard existed. Data were tested for normality using a Shapiro-Wilks test and are presented using medians (interquartile range [IQR]). Statistical analysis was conducted using SPSS version 25 (IBM, New York, United States). P < 0.05 was considered statistically significant.

Hospitals were ranked by quartiles according to their median performance against JAG standards. These fall broadly into training activity (training list per trainee, points per training list and DOPS completed per 200 procedures) and training performance (upper and lower gastrointestinal endoscopy KPIs). The top-performing unit was awarded an additional point, and the lowest scoring unit received a score of 0, creating a scale of 0 to 5. Points per training list scores were adapted, because higher quartiles, representing more points per training list, did not correspond with better training performance. A JAG standard of 8 points equated to a score of 5 and graduated to a minimum score of 0 for maximal over- and under- booked lists. Scores were then used to generate a composite index with overall grading (A + to D) based on quartiles, adapted from Hopkins et al, comparing hospital surgical training unit performance in Wales [7].

A radar chart was created for each individual hospital unit, each arm representing the zero to five scale of each metric. These were plotted in Excel (Microsoft, Redmond, Washington, United States) to compare individual training units visually. Percentage coverage was calculated using standard trigonometry.

Results

Quality performance metric median values related to individual hospital units can be found in [Table 1], and further classified by hospital according to local health boards in [Table 2]. Median training list allocation per trainee was 13 (10–17), with only two hospital units (D17, D18) meeting the JAG standard of 20 training lists per year (P = 0.001, effect size d = –0.56). The JAG recommendation that training lists be adjusted to eight points was achieved by one hospital (D17). Median points per list were 11 (5–18, P = 0.064, d = 0.31). Six hospitals under-utilized training opportunities (D1, 7, 11, 12, 13, 14), and 11 over-booked lists, risking inappropriate training environments (T3, D2, 4, 5, 6, 8, 9,10,15,16,18). Median annual total procedure numbers were 1395 (465–2365), amounting to 115 (39–214) procedures per trainee. Dividing the total DOPS per unit by the number of trainees per unit; resulting in median three (1–6) DOPS per trainee per year.

Table 1 Endoscopy training quality performance metrics related to individual hospital units.

Metric	Median (IQR)	Mean (SD)	JAG standard	P value (effect size)
Annual training procedures	1395 (456–2365)	1607 (1270)
EGD	732 (237–1275)	811 (634)
Colonoscopy	386 (176–837)	517 (434)
Flexible sigmoidoscopy	178 (50–363)	279 (290)
No. training lists	143 (92–239)	156 (88)
No. trainees	12 (7–16)	12 (7)
No. trainers	11 (6–18)	11 (6)
Trainee:trainer	0.8 (0.7–1.3)	0.9 (0.4)
Training list per trainee	13 (10–17)	13 (6)	20	0.001* (d = -0.56)
Annual number of procedures per trainee	115 (39–214)	145 (140)
Points per adjusted training list	11 (5–18)	12 (8)	8	0.064 (d = 0.31)
DOPS competed per trainee	3 (1–6)	4 (3)
DOPS completed per trainer	4 (1–7)	4 (3)
DOPS completed per list	0.2 (0.1–0.4)	0.3 (0.2)
DOPS completed per 200 procedures	6 (2–12)	7 (6)	20	< 0.001* (d = -0.61)
D2 Intubation	93 % (91–96)	93 % (4)	> 95 %	0.068 (d = -0.30)
J maneuver	94 % (90–96)	93 % (4)	> 95 %	0.149 (d = -0.24)
Cecal Intubation	82 % (72–90)	80 % (15)	> 90 %	0.011* (d = -0.44)
Polyp detection rate	25 % (18–34)	25 % (10)	> 20 %	0.035* (d = 0.35)

IQR, interquartile range; SD, standard deviation; JAG, Joint Advisory Group on Gastrointestinal Endoscopy; EGD, esophagogastroduodenoscopy; DOPS, direct observation of procedural skills.

Table 2 Endoscopy training quality performance metrics related to local health boards.

Metric	Median (range)	Mean (SD)	JAG standard	P value (effect size)
Annual training procedure	3866 (2035–7865)	4817 (3657)
EGD	1510 (1510–3098)	1968 (1957)
Colonoscopy	1038 (704–1750)	1263 (905)
Flexible sigmoidoscopy	631 (214–2734)	1248 (1346)
Number of training lists	490 (356–580)	467 (140)
No. trainees	33 (23–48)	35 (15)
No. trainers	37 (21–46)	34 (13)
Trainee:trainer	1 (0.8–1.5)	1 (0.31)
Training list per trainee	13 (12–18)	14 (3)	20	0.028 (d = -0.64)
Annual number of procedures per trainee	117 (64–193)	135 (96)
Points per adjusted training list	10 (8–13)	11 (6)	8	0.249 (d = 0.33)
DOPS competed per trainee	3 (2–6)	4 (3)
DOPS completed per trainer	3 (2–7)	4 (3)
DOPS completed per list	0.24 (0.16–0.42)	0.28 (0.15)
DOPS completed per 200 procedures	9 (2–11)	8 (4)	20	0.028 (d = -0.64)
D2 Intubation	94 % (91–96)	93 % (3)	> 95 %	0.345 (d = -0.27)
J maneuver	94 % (92–97)	94 % (3)	> 95 %	0.249 (d = -0.33)
Cecal Intubation	85 % (81–90)	78 % (17)	> 90 %	0.075 (d = -0.51)
Polyp detection rate	24 % (21–30)	27 % (6)	> 20 %	0.028 (d = 0.64)

IQR, interquartile range; SD, standard deviation; JAG, Joint Advisory Group on Gastrointestinal Endoscopy; EGD, esophagogastroduodenoscopy; DOPS, direct observation of procedural skills.

Median DOPS per 200 procedures was six (2–12); significantly fewer than the JAG target (P < 0.001, d = –0.61), with only one hospital (D11) achieving the standard.

Hospitals were ranked by quartiles according to their median performance against KPI JAG standards ([Fig. 1]). Individual hospital performance can be found in Supplementary Table 1.

Fig. 1 Key performance indicators (KPIs) per hospital unit (ranked by quartiles). Bars represent median with interquartile range. Shaded section represents performance meeting/exceeding Joint Advisory Group on Gastrointestinal Endoscopy standard for each KPI.

Esophagogastroduodenoscopy KPIs

D2 intubation and J maneuver were 93 % (91–96) and 94 % (90–96) respectively, similar to the JAG standard of 95 % (P = 0.068 d = –0.30; P = 0.149 d = –0.24).

Colonoscopy KPIs

Caecal intubation was 82 % (72–90); 14 hospitals achieved lower rates than the JAG standard (90 %, P = 0.011, d = –0.44). polyp detection rate (PDR) was 24 % (21–30); 13 hospitals exceeded the JAG standard (20 %, P = 0.035, d = 0.35).

Compound-level unit quality analysis

Compound-level scores were calculated for all hospitals and can be found in [Fig. 2]. The median score was 17 (14–20), with a mean of 17 (standard deviation (SD) 5). Radar charts demonstrating the relative performance of the best versus the worst units can be found in [Fig. 3]. Median radar coverage was 22.9% (16.4 %–34.2 %), mean 27.1 % (SD 13.8 %).

Fig. 2 Compound scores related to hospital unit. D, district general hospital; T, tertiary hospital; red, quartile 1; orange, quartile 2; yellow, quartile 3; green, quartile 4.

Fig. 3 Radar chart of lowest and highest performing unit.

Discussion

The age of deliberate accountability has arrived, of bonus for measured performance, and of faith in the value of publishing outcomes to guarantee clarity. This is the first study to provide objective comparative data regarding hospital unit level endoscopy training quality metrics within a UK Statutory Education Body. The salient findings related to KPI variation and consequently training deficit.

Esophagogastroduodenoscopy (EGD) KPIs with regard to D2 intubation and J maneuver were 96 % in upper quartile units compared with 91 % and 90 % in the lower quartile: Colonoscopy KPIs with regards to cecal Intubation and PDR were 90 % and 30 % in upper quartile units compared with 72 % and 21 % in lower quartile units, translating to training metric radar plot areas that differed four-fold.

The surgical adage and prior common practice of “see one, do one, teach one” is long gone and fails contemporary scrutiny and outcome metrics in the modern medical world of audit, measured accountability, and patient safety and quality of care. With the European working time directive, reduction in working hours and “shape of training” shortening the length of specialist training [8], it is more important than ever to train effectively, and to recognize competency-based proficiency rather than arbitrary minimum thresholds. A recent review of training provision for Gastroenterology trainees in Wales, revealed a two-fold variation across health boards, at a rate inadequate to achieve full certification in gastroscopy and colonoscopy in the “shape of training” era [9]. Indeed, JAG considers both measures, with certification awarded following completion of competency-based assessments in addition to a minimum level of indicative certified numbers of procedures [10].

Dissatisfaction related to endoscopy training is most commonly attributed to poor access to training lists, with training progression heavily influenced by hospital placement on-call rotas and other general medical responsibilities [11] This issue amplified with respect to surgical trainees in view of competing scheduled elective surgical demands [4] [12] [13], resulting in as few as 18 % of Colorectal trainees achieving the minimum number of colonoscopy procedures by the end of training [14]. The most recent UK cross-specialty survey reinforced this, highlighting again a discrepancy in training, with 38.9 % of surgical trainees accessing one training list per week, and 70 % of gastroenterology trainees having two or more scheduled endoscopy lists per week [15]. Furthermore, the UK JAG census reported a 12 % to 15 % increase in service pressures between 2017 and 2019, with a clear adverse effect on training: dedicated training list numbers falling from 76 % in 2017 to 52 % in 2019 [16]. The introduction of the Clinical Endoscopy Training Programme in 2016 aimed to improve service provision and has increased training list requirement two-fold for clinical nurse endoscopists [17]. Hospital size and teaching status does not reflect the output through an endoscopy unit, with smaller hospitals potentially better equipped to train in endoscopy given the lack of alternative competing interests. In an increasingly competitive arena for training resource, increased focus on the practicalities of training provision for all groups of endoscopy trainees should ensure equity.

Quality metrics related to endoscopy training utilizing surrogate markers of hospital performance are conspicuous by their absence. JAG considers factors such as; simulation, course attendance, lifetime procedure numbers, and frequency of training and non-training lists, as markers of training quality [12]. What this study adds is training activity metrics, and in particular: numbers of training lists, adjustments to training needs, completion of satisfactory summative DOPS assessments, and training performance KPIs. Hospital, compound-level surgical training quality performance metrics illustrated by means of radar chart plots have been reported previously; specifically, to compare core surgical training quality related to hospital units in the Wales SEB. Important performance training metric variation was apparent for 6-month hospital placements, with trainees in upper quartile units achieving on average, 20 % more operations, 40 % more WBAs, and one more peer reviewed publication, than trainees in lower quartile units [7]. Compound-level surgical training and endoscopy training quality performance metrics were not consistent: with one bottom-quartile performing unit for surgical training in the top quartile for endoscopy training; seven units featuring in the same quartile, and seven district general hospitals providing endoscopy training without surgical training.

This study has a number of inherent limitations. The data pertain to a solitary UK SEB, and therefore, they must be interpreted with caution. Individual trainee outcomes were not assessed, and as a result, individual hospital unit training quality was unable to be directly compared based on trainee performance. Furthermore, relative learning curve trajectories of these trainees could not be determined. Factors affecting the learning curve in colonoscopy have been previously described [10], but the learning curve trajectory was not studied. In relation to six general surgical indicative procedures, the trajectory varied 5-fold [18]: endoscopy learning curve trajectories are worthy of further research but are arguably likely to vary similarly.

Moreover, this was a retrospective analysis of prospectively collected data from all cross-specialty trainees who recorded training activity on JETS: measuring competence related to trainees’ self-perceived and reported KPIs also has limitations. Surgical trainees are allocated rotations within individual hospital units; however, training opportunities may present themselves across sites within local health boards. It is possible that trainees and, or trainers are registered at more than one hospital which may risk duplication within the JETS database. Training lists and numbers of DOPS completed per 200 procedures were calculated related to individual trainees, and crude data inaccuracy could influence results. It is possible that endoscopy services may be unevenly distributed between hospitals within a local health board, this should be considered when interpreting the results and planning trainee allocation.

Through initiating GRS, JAG attempts to define clearly what a training list should be: with a qualified trainer and an appropriate case mix and number. However, it is known that some “training lists” exist that do not meet the definition. While JAG recommend adjusting a training list to 8 points and reflect an appropriate case mix, it should be recognized that this broad recommendation requires tailoring to the need of the individual trainee. An endoscopist at the start of their endoscopy training may initially require less than 8 points, while those more senior, approaching certification, should be performing at an independent level and therefore a 10- to 12-point training list would be more appropriate. Considering this recommendation as a fixed standard in this study does not reflect the need for adaptability and can only be generalized given the lack of specific trainee outcomes or stages of training. Furthermore, trainers and trainees alike highlight the importance of ad hoc lists, especially as trainees increase their experience and become more independent. This study does not capture these lists, in which opportunistic training occurs.

Recognizing the variation in training quality across hospital units is imperative to maximize available opportunities, however this study strengthens existing evidence that poor access to training lists remains a significant problem. Endoscopy training fellowships have been suggested as one option to compensate for these deficiencies [9]. In addition, the SPRINT program: Structured Programme for Induction and Training, an existing initiative to improve the training delivery in Wales, incorporates simulator and lesion recognition training, with endoscopic non-technical skills, and has been reported to shorten time to reach 200 procedures [19]. With the evolution of high-fidelity virtual reality simulators, this is an area requiring further research to develop and validate a structured simulator-based training curriculum as an adjunct to hands-on training.

Conclusions

Important disparities in hospital endoscopy unit performance were observed and disguised by the cloak of clinical pressures currently prevalent in the NHS. Compound hospital training quality varied three-fold. Trainees, trainers and training program directors alike should be aware of such data when planning educational programs, so that the quality of endoscopy training can be focused and optimized. JAG now considers simulation to be an important and integral marker of training. Adding simulation to the training armamentarium should be urgently recognized as a paramount constituent of the recovery-phase of COVID-19 training catch-up strategy, in order to overcome rationed front-line clinical training opportunities and also to address the pressing clinical service back-log of urgent suspected cancer referrals. Development of a nationally agreed and accredited curriculum allied to endoscopic virtual reality haptic feedback will be key to recovery and improved endoscopy training.

• References

• 1 Barton JR, Corbett S, Van Der Vleuten CP. The validity and reliability of a Direct Observation of Procedural Skills assessment tool: Assessing colonoscopic skills of senior endoscopists. Gastrointest Endosc 2012; 75: 591-597
• 2 Gupta S, Anderson J, Bhandari P. et al. Development and validation of a novel method for assessing competency in polypectomy: Direct observation of polypectomy skills. Gastrointest Endosc 2011; 73: 1232-1239
• 3 Joint Advisory Group on GI Endoscopy. Formative DOPS: Diagnostic upper gastrointestinal endoscopy (OGD). https://www.thejag.org.uk/AboutUs/DownloadCentre.aspx
• 4 Jones RP, Stylianides NA, Robertson AG. et al. National survey on endoscopy training in the UK. Ann Royal Coll Surg England 2015; 97: 386-389
• 5 Health Education England. Clinical Endoscopist Training Programme. https://www.hee.nhs.uk/our-work/endoscopy/clinical-endoscopist-training-programme
• 6 Rutter MD, Brookes M, Lee TJ. et al. Impact of the COVID-19 pandemic on UK endoscopic activity and cancer detection: a national endoscopy database analysis. Gut 2020;
• 7 Hopkins L, Robinson D, Brown C. et al. Hospital compound level surgical training quality performance metrics: more than a score?. Bull Royal Coll Surg Engl 2020; 102: e1-e8
• 8 Clough J, Fitzpatrick M, Harvey P. et al. Shape of training review: An impact assessment for UK gastroenterology trainees. Frontline Gastroenterol 2019; 10: 356-363
• 9 Hawkes N, Karunaratne D, Phillips A. P396 Differences in endoscopy training rates in Wales – implications in the shape of training era. Gut 2021; 70: A245
• 10 Ward ST, Mohammed MA, Walt R. et al. An analysis of the learning curve to achieve competency at colonoscopy using the JETS database. Gut 2014; 63: 1746-1754
• 11 Biswas S, Alrubaiy L, China L. et al. Trends in UK endoscopy training in the BsG trainees’ national survey and strategic planning for the future. Frontline Gastroenterol 2018; 9: 200-207
• 12 Hammond JS, Watson NFS, Lund JN. et al. Surgical endoscopy training: the Joint Advisory Group on gastrointestinal endoscopy national review. Frontline Gastroenterol 2013; 4: 20-24
• 13 Patel K, Ward S, Gash K. et al. Prospective cohort study of surgical trainee experience of access to gastrointestinal endoscopy training in the UK and Ireland. Int J Surgery 2019; 63: 113-116
• 14 Malik KI, Siau K, Dunckley P. et al. Colorectal trainees in the UK struggle to meet JAG certification standards in colonoscopy by the end of their training. Colorectal Dis 2019; 21: 715-722
• 15 Ratcliffe E, Subramaniam S, Ngu WS. et al. Endoscopy training in the UK pre-COVID-19 environment: A multidisciplinary survey of endoscopy training and the experience of reciprocal feedback. Frontline Gastroenterol 2021;
• 16 Ravindran S, Bassett P, Shaw T. et al. National census of UK endoscopy services in 2019. Frontline Gastroenterol 2020; 12: 1-10
• 17 Siau K, Anderson J, Valori R. et al. Certification of UK gastrointestinal endoscopists and variations between trainee specialties: results from the JETS e-portfolio. Endosc Int Open 2019; 7: E551-E560
• 18 Brown C, Abdelrahman T, Patel N. et al. Operative learning curve trajectory in a cohort of surgical trainees. British Journal of Surgery 2017; 104: 1405-1411
• 19 Hawkes N, Turner J, Hurley J. et al. Accelerated training in upper GI endoscopy-an analysis of sprint programme outcomes. United Eur Gastroenterol J 2015; 3: A368-A369