Work-Based Assessments in Higher General Surgical Training Program: A Mixed Methods Study Exploring Trainers' and Trainees' Views and Experiences

Abstract

Introduction In the United Kingdom, work-based assessments (WBAs) including procedure-based assessments (PBAs), case-based discussions (CBDs), clinical evaluation exercises (CEXs), and direct observation of procedural skills (DOPS) have been used in Higher General Surgical Training Program (HGSTP) since the introduction of Modernising Medical Careers. Although the Intercollegiate Surgical Curriculum Project states that they should be used for the formative development of trainees using feedback and reflection, there is no study to look at the perception of their usefulness and barriers in using them, particularly in HGSTP. The aim of this study is to investigate trainer's and trainee's perception of their usefulness, barriers in using them, and way forward for their improvement in HGSTP.

Methods This was a mixed method study. In phase I, after ethics committee approval, an online survey was sent to 83 trainers and 104 trainees, with a response rate of 33 and 37%, respectively, using Online Surveys (formerly Bristol Online Survey) from July 2018 to December 2018. After analysis of this result, in phase II, semistructured interviews were conducted with five trainees and five trainers who had volunteered to take part from phase I. Thematic analysis was performed to develop overarching themes.

Results For professional formative development, 15% of the trainers and 53% of the trainees felt that WBAs had a low value. Among 4 WBAs—CEX, CBD, PBA, and DOPS—PBA was thought to be the most useful WBA by 52% trainers and 74% trainees.

More trainers than trainees felt that it was being used as a formative tool (33 vs. 16%). The total number of WBAs thought to be required was between 20 and 40 per year, with 46% of the trainers and 53% of the trainees preferring these numbers.

The thematic analysis generated four themes with subthemes in each: theme 1, “factors affecting usefulness,” including the mode of validation, trainer/trainee engagement, and time spent in validating; theme 2, “doubt on utility” due to doubt on validity and being used as a tick-box exercise; theme 3, “pitfalls/difficulties” due to lack of time to validate, late validation, e-mail rather than face-to-face validation, trainer and trainee behavior, variability in feedback given, and emphasis on number than quality; and theme 4, “improvement strategies.”

Conclusions The WBAs are not being used in a way they are supposed to be used. The perception of educational impact (Kirkpatrick levels 1 and 2) by trainers was more optimistic than by trainees. Improvements can be made by giving/finding more time, trainer training, more face-to-face validation, and better trainer trainee interactions.

Keywords

work-based assessments - surgery training - procedure-based assessments - case-based discussion - clinical evaluation exercises - direct observation of procedural skills

• References

• 1 Bannon M. What's happening in postgraduate medical education?. Arch Dis Child 2006; 91 (01) 68-70
• 2 Modernising Medical Careers. Medical specialty training (England). www.mmc.nhs.uk . Accessed December 3, 2018
• 3 Swanwick T, Chana N. Workplace-based assessment. Br J Hosp Med (Lond) 2009; 70 (05) 290-293
• 4 General Medical Council. Training pathways: analysis of the transition from the foundation programme to the next stage of training. https://www.gmc-uk.org/-/media/documents/dc10999-evi-training-pathways-analysis-of-transition-from-foundation-to-next-stage-of-train-74522826.pdf . Accessed July 23, 2018
• 5 Van Der Vleuten CP. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract 1996; 1 (01) 41-67
• 6 Kirkpatrick D. . Evaluation of training. In: Craig RL, Bittel LR, eds. Training and Development Handbook. New York, NY: McGraw-Hill; 1967: 87-112
• 7 Saedon H, Salleh S, Balakrishnan A, Imray CH, Saedon M. The role of feedback in improving the effectiveness of workplace based assessments: a systematic review. BMC Med Educ 2012; 12: 25
• 8 Matthew A, Beard JD, Bussey M. An evaluation of the use of direct observation of procedural skills in the intercollegiate surgical curriculum programme. Ann R Coll Surg Engl (Suppl) 2014; 96: e10-e13
• 9 Shalhoub J, Marshall DC, Ippolito K. Perspectives on procedure-based assessments: a thematic analysis of semistructured interviews with 10 UK surgical trainees. BMJ Open 2017 ;7(3):e013417
• 10 James K, Cross K, Lucarotti ME, Fowler AL, Cook TA. Undertaking procedure-based assessment is feasible in clinical practice. Ann R Coll Surg Engl 2009; 91 (02) 110-112
• 11 Joshi MK, Singh T, Badyal DK. Acceptability and feasibility of mini-clinical evaluation exercise as a formative assessment tool for workplace-based assessment for surgical postgraduate students. J Postgrad Med 2017; 63 (02) 100-105
• 12 Phillips A, Lim J, Madhavan A, Macafee D. Case-based discussions: UK surgical trainee perceptions. Clin Teach 2016; 13 (03) 207-212
• 13 Pereira EA, Dean BJ. British surgeons' experiences of mandatory online workplace-based assessment. J R Soc Med 2009; 102 (07) 287-293
• 14 Pereira EA, Dean BJ. British surgeons' experiences of a mandatory online workplace based assessment portfolio resurveyed three years on. J Surg Educ 2013; 70 (01) 59-67
• 15 Philips AW, Jones AE. The validity and reliability of work place based assessments in surgical training. Bull R Coll Surg Engl 2015; 97: e19-e23
• 16 Taylor PC, Medina M. Educational research paradigms form positivism to pluralism. Coll Res J 2011; 1 (01) 1-16
• 17 Lee AS. Integrating positivist and interpretive approaches to organizational research. Organ Sci 1991; 2 (04) 342-365
• 18 Illing J. . Thinking about research: theoretical perspectives, ethics and scholarship. In: Swanwick T, ed. Understanding Medical Education: Evidence, Theory and Practice. 2nd ed. Oxford: ASME and Wiley-Blackwell; 2013: 331-348
• 19 Bunniss S, Kelly DR. Research paradigms in medical education research. Med Educ 2010; 44 (04) 358-366
• 20 Schifferdecker KE, Reed VA. Using mixed methods research in medical education: basic guidelines for researchers. Med Educ 2009; 43 (07) 637-644
• 21 Boynton PM, Greenhalgh T. Selecting, designing, and developing your questionnaire. BMJ 2004; 328 (7451): 1312-1315
• 22 Burford B, Hesketh A, Wakeling J. , et al. Asking the right questions and getting meaningful responses: 12 tips on developing and administering a questionnaire survey for healthcare professionals. Med Teach 2009; 31 (03) 207-211
• 23 Beard JD, Marriott J, Purdie H, Crossley J. Assessing the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology. Health Technol Assess 2011; 15 (01) i –xxi, 1–162
• 24 Maudsley G. Mixing it but not mixed-up: mixed methods research in medical education (a critical narrative review). Med Teach 2011; 33 (02) e92-e104
• 25 Lingard L, Kennedy TJ. Qualitative research methods in medical education. In: Swanwick T, ed. Understanding Medical Education: Evidence, Theory and Practice. Oxford: ASME and Wiley-Blackwell; 2014: 371-384
• 26 Gibbs G. Data preparation. In: Analysing Qualitative Data. London: Sage; 2007: 10-22
• 27 Patton MQ. Qualitative Evaluation and Research Methods. 2nd ed. Newbury Park, CA: Sage; 1990
• 28 Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In Bryman A, Burgess RG, eds. Analyzing Qualitative Data. London: Routledge; 1994: 173-194
• 29 Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006; 3 (02) 77-101
• 30 Aryal KR, Downey S, Praseedom R. Current practice of work based assessments in UK higher general surgical training. Ann R Coll Surg Engl [In Press]
• 31 Pentlow A, Field J. The educational value of work-based assessments: a survey of orthopaedic trainees and their consultant trainers. J Surg Simul. 2015; 2: 60-67
• 32 Hunter AR, Baird EJ, Reed MR. Procedure-based assessments in trauma and orthopaedic training--the trainees' perspective. Med Teach 2015; 37 (05) 444-449
• 33 Bindal T, Wall D, Goodyear HM. Trainee doctors' views on workplace-based assessments: are they just a tick box exercise?. Med Teach 2011; 33 (11) 919-927
• 34 Boursicot K, Etheridge L, Setna Z. , et al. Performance in assessment: consensus statement and recommendations from the Ottawa conference. Med Teach 2011; 33 (05) 370-383
• 35 Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VS. Effect of rater training on reliability and accuracy of mini-CEX scores: a randomized, controlled trial. J Gen Intern Med 2009; 24 (01) 74-79
• 36 Massie J, Ali JM. Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings. Adv Health Sci Educ Theory Pract 2016; 21 (02) 455-473
• 37 Eardley I, Bussey M, Woodthorpe A, Munsch C, Beard J. Workplace-based assessment in surgical training: experiences from the Intercollegiate Surgical Curriculum Programme. ANZ J Surg 2013; 83 (06) 448-453
• 38 Basu I, Parvizi S, Chin K. The perception of online work-based assessments. Clin Teach 2013; 10 (02) 73-77
• 39 McKavanagh P, Smyth A, Carragher A. Hospital consultants and workplace based assessments: how foundation doctors view these educational interactions?. Postgrad Med J 2012; 88 (1037): 119-124
• 40 Gaunt A, Patel A, Rusius V, Royle TJ, Markham DH, Pawlikowska T. 'Playing the game': how do surgical trainees seek feedback using workplace-based assessment?. Med Educ 2017; 51 (09) 953-962
• 41 ten Cate O, Scheele F. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice?. Acad Med 2007; 82 (06) 542-547
• 42 van Loon KA, Driessen EW, Teunissen PW, Scheele F. Experiences with EPAs, potential benefits and pitfalls. Med Teach 2014; 36 (08) 698-702
• 43 Gill P, Stewart K, Treasure E, Chadwick B. Methods of data collection in qualitative research: interviews and focus groups. Br Dent J 2008; 204 (06) 291-295