Download PDF
Endoscopy 2021; 53(12): 1235-1245
DOI: 10.1055/a-1352-7293
Systematic review

Colonoscopy competence assessment tools: a systematic review of validity evidence

Rishad Khan 
 1   Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, Canada
,
Eric Zheng
 1   Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, Canada
,
Sachin B. Wani
 2   Division of Gastroenterology and Hepatology, University of Colorado Anschutz Medical Campus, Aurora, Colorado, USA
,
Michael A. Scaffidi 
 3   Faculty of Medicine, Queen’s University, Kingston, Canada
 4   Division of Gastroenterology, St. Michael’s Hospital, Toronto, Canada
,
Thurarshen Jeyalingam
 1   Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, Canada
 5   The Wilson Centre, University of Toronto, Toronto, Canada
,
Nikko Gimpaya
 4   Division of Gastroenterology, St. Michael’s Hospital, Toronto, Canada
,
John T. Anderson 
 6   Joint Advisory Group on Gastrointestinal Endoscopy, Royal College of Physicians, London, UK
 7   Department of Gastroenterology, Gloucestershire Hospitals NHS Foundation Trust, Gloucestershire, UK
,
Samir C. Grover
 1   Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, Canada
 4   Division of Gastroenterology, St. Michael’s Hospital, Toronto, Canada
 8   Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Canada
,
Graham McCreath
 9   SickKids Research and Lerning Institutes, The Hospital for Sick Children, Toronto, Canada
,
Catharine M. Walsh 
 5   The Wilson Centre, University of Toronto, Toronto, Canada
 9   SickKids Research and Lerning Institutes, The Hospital for Sick Children, Toronto, Canada
10   Division of Gastroenterology, Hepatology, and Nutrition, The Hospital for Sick Children, Toronto, Canada
11   Department of Paediatrics, Faculty of Medicine, University of Toronto, Toronto, Canada
› Author Affiliations
› Further Information
Also available at
    Permissions and Reprints

Referred to by:

Author commentary on Rishad Khan et al.Endoscopy 2021; 53(12): v60-v60
DOI: 10.1055/a-1391-1545

Abstract

Background Assessment tools are essential for endoscopy training, being required to support feedback provision, optimize learner capabilities, and document competence. We aimed to evaluate the strength of validity evidence that supports the available colonoscopy direct observation assessment tools using the unified framework of validity.

Methods We systematically searched five databases for studies investigating colonoscopy direct observation assessment tools from inception until 8 April 2020. We extracted data outlining validity evidence (content, response process, internal structure, relations to other variables, and consequences) from the five sources and graded the degree of evidence, with a maximum score of 15. We assessed educational utility using an Accreditation Council for Graduate Medical Education framework and methodological quality using the Medical Education Research Quality Instrument (MERSQI).

Results From 10 841 records, we identified 27 studies representing 13 assessment tools (10 adult, 2 pediatric, 1 both). All tools assessed technical skills, while 10 each assessed cognitive and integrative skills. Validity evidence scores ranged from 1–15. The Assessment of Competency in Endoscopy (ACE) tool, the Direct Observation of Procedural Skills (DOPS) tool, and the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) had the strongest validity evidence, with scores of 13, 15, and 14, respectively. Most tools were easy to use and interpret, and required minimal resources. MERSQI scores ranged from 9.5–11.5 (maximum score 14.5).

Conclusions The ACE, DOPS, and GiECAT have strong validity evidence compared with other assessments. Future studies should identify barriers to widespread implementation and report on the use of these tools in credentialing examinations.

Fig. 1s, Tables 1s–6s, Appendices 1s–13s



Publication History

Received: 13 September 2020

Accepted after revision: 13 January 2021

Accepted Manuscript online:
13 January 2021

Article published online:
16 March 2021

© 2021. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Faigel DO, Baron T, Lewis B. et al. Ensuring competence in endoscopy. ASGE policy and procedures manual for gastrointestinal endoscopy: guidelines for training and practice. 2005 Accessed: 24 June 2020 www.asge.org/docs/default-source/education/practice_guidelines/doc-competence.pdf?sfvrsn=6
    Google Scholar
  • 2 Walsh CM. In-training gastrointestinal endoscopy competency assessment tools: types of tools, validation and impact. Best Pract Res Clin Gastroenterol 2016; 30: 357-374
    CrossrefPubMedGoogle Scholar
  • 3 Dafnis G, Granath F, Påhlman L. et al. The impact of endoscopists’ experience and learning curves and interendoscopist variation on colonoscopy completion rates. Endoscopy 2001; 33: 511-515
    Article in Thieme ConnectPubMedGoogle Scholar
  • 4 Matharoo M, Haycock A, Sevdalis N. et al. A prospective study of patient safety incidents in gastrointestinal endoscopy. Endosc Int Open 2017; 5: E83-E89
    Article in Thieme ConnectPubMedGoogle Scholar
  • 5 Lockyer J, Carraccio C, Chan M-K. et al. Core principles of assessment in competency-based medical education. Med Teach 2017; 39: 609-616
    CrossrefPubMedGoogle Scholar
  • 6 Govaerts MJ, van der Vleuten CP, Schuwirth LW. et al. Broadening perspectives on clinical performance assessment: rethinking the nature of in-training assessment. Adv Health Sci Educ 2007; 12: 239-260
    CrossrefPubMedGoogle Scholar
  • 7 Frank JR, Snell LS, Cate OT. et al. Competency-based medical education: theory to practice. Med Teach 2010; 32: 638-645
    CrossrefPubMedGoogle Scholar
  • 8 Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006; 119: 166.e7-166.e16
    CrossrefPubMedGoogle Scholar
  • 9 Messick S. Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. Am Psychol 1995; 50: 741
    CrossrefPubMedGoogle Scholar
  • 10 Moher D, Liberati A, Tetzlaff J. et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009; 6: e1000097
    CrossrefPubMedGoogle Scholar
  • 11 Shea BJ, Reeves BC, Wells G. et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017; 358: j4008
    Google Scholar
  • 12 Preisler L, Svendsen MBS, Svendsen LB. et al. Methods for certification in colonoscopy – a systematic review. Scand J Gastroenterol 2018; 53: 350-358
    CrossrefPubMedGoogle Scholar
  • 13 Watson PF, Petrie A. Method agreement analysis: a review of correct methodology. Theriogenology 2010; 73: 1167-1179
    CrossrefPubMedGoogle Scholar
  • 14 Cook DA, Brydges R, Zendejas B. et al. Technology-enhanced simulation to assess health professionals: a systematic review of validity evidence, research methods, and reporting quality. Acad Med 2013; 88: 872-883
    CrossrefPubMedGoogle Scholar
  • 15 Beckman TJ, Cook DA, Mandrekar JN. What is the validity evidence for assessments of clinical teaching?. J Gen Intern Med 2005; 20: 1159-1164
    CrossrefPubMedGoogle Scholar
  • 16 Ghaderi I, Manji F, Park YS. et al. Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg 2015; 261: 251-262
    CrossrefPubMedGoogle Scholar
  • 17 Swing SR, Clyman SG, Holmboe ES. et al. Advancing resident assessment in graduate medical education. J Grad Med Educ 2009; 1: 278-286
    CrossrefPubMedGoogle Scholar
  • 18 Cook DA, Reed DA. Appraising the quality of medical education research methods: the medical education research study quality instrument and the Newcastle–Ottawa scale-education. Acad Med 2015; 90: 1067-1076
    CrossrefPubMedGoogle Scholar
  • 19 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159-174
    CrossrefPubMedGoogle Scholar
  • 20 Sedlack RE. The Mayo Colonoscopy Skills Assessment Tool: validation of a unique instrument to assess colonoscopy skills in trainees. Gastrointest Endosc 2010; 72: 1125-1133
    CrossrefPubMedGoogle Scholar
  • 21 Sedlack RE, Coyle WJ, Obstein KL. et al. ASGE’s assessment of competency in endoscopy evaluation tools for colonoscopy and EGD. Gastrointest Endosc 2014; 79: 1-7
    CrossrefPubMedGoogle Scholar
  • 22 Sedlack RE, Coyle WJ. Assessment of competency in endoscopy: establishing and validating generalizable competency benchmarks for colonoscopy. Gastrointest Endosc 2016; 83: 516-523
    CrossrefPubMedGoogle Scholar
  • 23 Han S, Obuch JC, Duloy AM. et al. A prospective multicenter study evaluating endoscopy competence among gastroenterology trainees in the era of the next accreditation system. Acad Med 2020; 95: 283-292
    CrossrefPubMedGoogle Scholar
  • 24 Boyle E, Al-Akash M, Patchett S. et al. Towards continuous improvement of endoscopy standards: validation of a colonoscopy assessment form. Colorectal Dis 2012; 14: 1126-1131
    Google Scholar
  • 25 Barton JR, Corbett S, van der Vleuten CP. et al. The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists. Gastrointest Endosc 2012; 75: 591-597
    CrossrefPubMedGoogle Scholar
  • 26 Siau K, Dunckley P, Valori R. et al. Changes in scoring of Direct Observation of Procedural Skills (DOPS) forms and the impact on competence assessment. Endoscopy 2018; 50: 770-778
    Article in Thieme ConnectPubMedGoogle Scholar
  • 27 Siau K, Crossley J, Dunckley P. et al. Colonoscopy Direct Observation of Procedural Skills assessment tool for evaluating competency development during training. Am J Gastroenterol 2020; 115: 234-243
    CrossrefPubMedGoogle Scholar
  • 28 Siau K, Levi R, Iacucci M. et al. Paediatric colonoscopy Direct Observation of Procedural Skills: evidence of validity and competency development. J Pediatr Gastroenterol Nutr 2019; 69: 18-23
    CrossrefPubMedGoogle Scholar
  • 29 Vassiliou MC, Kaneva PA, Poulose BK. et al. Global Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy. Surg Endosc 2010; 24: 1834-1841
    CrossrefPubMedGoogle Scholar
  • 30 Vassiliou MC, Kaneva PA, Poulose BK. et al. How should we establish the clinical case numbers required to achieve proficiency in flexible endoscopy?. Am J Surg 2010; 199: 121-125
    CrossrefPubMedGoogle Scholar
  • 31 Mueller CL, Kaneva P, Fried GM. et al. Colonoscopy performance correlates with scores on the FESTM manual skills test. Surg Endosc 2014; 28: 3081-3085
    CrossrefPubMedGoogle Scholar
  • 32 Sarker SK, Albrani T, Zaman A. et al. Procedural performance in gastrointestinal endoscopy: An assessment and self-appraisal tool. Am J Surg 2008; 196: 450-455
    CrossrefPubMedGoogle Scholar
  • 33 Walsh CM, Ling SC, Khanna N. et al. Gastrointestinal Endoscopy Competency Assessment Tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc 2014; 79: 798-807
    CrossrefPubMedGoogle Scholar
  • 34 Walsh CM, Ling SC, Khanna N. et al. Gastrointestinal Endoscopy Competency Assessment Tool: reliability and validity evidence. Gastrointest Endosc 2015; 81: 1417-1424
    CrossrefPubMedGoogle Scholar
  • 35 Scaffidi MA, Grover SC, Carnahan H. et al. A prospective comparison of live and video-based assessments of colonoscopy performance. Gastrointest Endosc 2018; 87: 766-775
    CrossrefPubMedGoogle Scholar
  • 36 Scaffidi MA, Grover SC, Carnahan H. et al. Impact of experience on self-assessment accuracy of clinical colonoscopy competence. Gastrointest Endosc 2018; 87: 827-836
    CrossrefPubMedGoogle Scholar
  • 37 Walsh CM, Ling SC, Walters TD. et al. Development of the Gastrointestinal Endoscopy Competency Assessment Tool for pediatric colonoscopy (GiECATKIDS). J Pediatr Gastroenterol Nutr 2014; 59: 480-486
    CrossrefPubMedGoogle Scholar
  • 38 Walsh CM, Ling SC, Mamula P. et al. The Gastrointestinal Endoscopy Competency Assessment Tool for pediatric colonoscopy. J Pediatr Gastroenterol Nutr 2015; 60: 474-480
    CrossrefPubMedGoogle Scholar
  • 39 Scaffidi MA, Khan R, Carnahan H. et al. Can pediatric endoscopists accurately assess their clinical competency? A comparison across skill levels. J Pediatr Gastroenterol Nutr 2019; 68: 311-317
    CrossrefPubMedGoogle Scholar
  • 40 Park J, MacRae H, Musselman LJ. et al. Randomized controlled trial of virtual reality simulator training: transfer to live patients. Am J Surg 2007; 194: 205-211
    CrossrefPubMedGoogle Scholar
  • 41 Shah SG, Thomas-Gibson S, Brooker JC. et al. Use of video and magnetic endoscope imaging for rating competence at colonoscopy: Validation of a measurement tool. Gastrointest Endosc 2002; 56: 568-573
    CrossrefPubMedGoogle Scholar
  • 42 Preisler L, Svendsen MBS, Nerup N. et al. Simulation-based training for colonoscopy: establishing criteria for competency. Medicine 2015; 94: e440
    CrossrefPubMedGoogle Scholar
  • 43 Konge L, Svendsen MBS, Preisler L. et al. Combining different methods improves assessment of competence in colonoscopy. Scand J Gastroenterol 2017; 52: 601-605
    CrossrefPubMedGoogle Scholar
  • 44 Koch AD, Haringsma J, Schoon EJ. et al. Competence measurement during colonoscopy training: the use of self-assessment of performance measures. Am J Gastroenterol 2012; 107: 971-975
    CrossrefPubMedGoogle Scholar
  • 45 Kumar NL, Kugener G, Perencevich ML. et al. The SAFE-T assessment tool: derivation and validation of a web-based application for point-of-care evaluation of gastroenterology fellow performance in colonoscopy. Gastrointest Endosc 2018; 87: 262-269
    CrossrefPubMedGoogle Scholar
  • 46 Kumar NL, Kugener G, Vemula A. et al. 214 Multi-center prospective validation study of the SAFE-T colonoscopy evaluation tool: a web-based smartphone application for evaluation of gastroenterology fellow performance in colonoscopy. Gastrointest Endosc 2019; 89: AB63
    CrossrefPubMedGoogle Scholar
  • 47 Mehta T, Dowler K, McKaig B. et al. Development and roll out of the JETS e-portfolio: a web based electronic portfolio for endoscopists. Frontline Gastroenterol 2011; 2: 35-42
    CrossrefPubMedGoogle Scholar
  • 48 Jørgensen M, Konge L, Subhi Y. Contrasting groups’ standard setting for consequences analysis in validity studies: reporting considerations. Adv Simul (Lond) 2018; 3: 5
    CrossrefPubMedGoogle Scholar
  • 49 Holmboe ES, Sherbino J, Long DM. et al. The role of assessment in competency-based medical education. Med Teach 2010; 32: 676-682
    CrossrefPubMedGoogle Scholar
  • 50 Patel SG, Keswani R, Elta G. et al. Status of competency-based medical education in endoscopy training: a nationwide survey of US ACGME-accredited gastroenterology training programs. Am J Gastroenterol 2015; 110: 956-962
    CrossrefPubMedGoogle Scholar