Global Evaluative Assessment of Robotic Skills in Endoscopy (GEARS-E): objective assessment tool for master and slave transluminal endoscopic robot

Abstract

Background and study aims The Master and Slave Transluminal Endoscopic Robot (MASTER) is a novel robotic endosurgical system with two operating arms that offer multiple degrees of freedom. We developed a new assessment tool, the Global Evaluative Assessment of Robotic Skills in Endoscopy (GEARS-E), derived from existing tools in laparoscopic and robotic surgery, and evaluated its utility in the performance of procedures using MASTER.

Methods This was a pilot study conducted in vivo and ex vivo on animals. Three operators (Novice-1, Novice-2 and Expert) performed a total of five endoscopic submucosal dissections (ESD) using MASTER. Novice operators had no MASTER experience and the expert had previously performed eight MASTER ESDs. Operator performance was assessed by four independent evaluators using GEARS-E, which has a maximum score of 25 for five domains representing various skill-related variables (depth perception, bimanual dexterity, efficiency, tissue handling and autonomy).

Results The mean global rating scores for Novice-1 first attempt, Novice-1 second attempt, Novice-2 first attempt, Novice-2 second attempt and Expertʼs cases were 13.0, 16.0, 13.3, 15.5, and 21.5, respectively. The mean scores of each of the five domains were statistically higher for the second attempts compared to the first attempts for both Novice-1 and Novice-2. The mean scores of each of the five domains for the Expertʼs case were consistently higher than those for the two novice operators in both their first and second attempts.

Conclusion Results using GEARS-E showed correlations between surgical experience and MASTER ESD. As an assessment tool for evaluation of surgical skills, GEARS-E has great potential for application in MASTER procedures.

Introduction

Endoscopic submucosal dissection (ESD) has become standard care for early gastrointestinal cancers [1] [2] [3]. Master and Slave Transluminal Endoscopic Robot (MASTER) is a novel robotic endosurgical system that has two operating arms equipped with multiple degrees of freedom, enabling complex maneuvers [4]. The feasibility of performing MASTER ESD has been shown in previous studies [5] [6] [7]. Going forward, the value of MASTER will need to be evaluated in terms of the user’s learning curve and training efficacy. Therefore, it is essential to develop an appropriate training program and a suitable skill assessment tool.

To provide an objective assessment tool in minimally invasive surgery, the Global Operative Assessment of Laparoscopic Skills (GOALS) was developed [8]. More recently, the Global Evaluative Assessment of Robotic Skills (GEARS), which was modeled after GOALS, was also developed [9]. There has not been an objective assessment tool developed for endoluminal surgical platforms such as MASTER. In this study, we explored use of a new scoring method, the Global Evaluative Assessment of Robotic Skills in Endoscopy (GEARS-E), which was modified from both GOALS and GEARS in evaluation of the learning curve of novice operators using MASTER.

Methods

Procedures

This study involved in vivo and ex vivo experiments. Participating operators performed ESD using MASTER on two adult female pigs, as well as on isolated pig tissues obtained from a commercial source. All experiments were conducted at Innoheart Pte Ltd with prior approval from the Institutional Animal Care and Use Committee. As for the in vivo experiments, the endoscope was introduced under general anesthesia and the operator telemanipulated the two robotic arms, one holding a cautery knife and the other a grasper, via cables that passed through the two channels of the endoscope. For the ex vivo experiments, the isolated colorectal segment was mounted on a stand to simulate its normal orientation. Then, an imaginary lesion was marked on the mucosa and MASTER ESD was performed. The pigs were euthanized after the procedures.

GEARS-E

In creating GEARS-E, we expanded the domains in GOALS and GEARS to encompass features unique to MASTER procedures ([Table 1]). Modeled after GOALS and GEARS, GEARS-E comprises six domains representing various skill-related variables (depth perception, bimanual dexterity, efficiency, tissue handling, autonomy and endoscope control). Assessment of each domain is done by scoring on a 5-point Likert scale with performance anchors at 1, 3 and 5. A rating of 1 denotes the lowest proficiency while 5 is the highest. As the focus of this study was on the novice’s training on robotic manipulators, we left the endoscope manipulation to a skilled endoscopist independently. To minimize bias from this, we excluded the sixth domain, endoscopic control, from the assessment. Therefore, the global rating score for this study was the sum of the scores for each of the remaining five domains, the maximum total score of which is 25.

Table 1 Six domains of GEARS-E.

Depth perception
1	2	3	4	5
Constantly overshoots target, wide swings, slow to correct		Some overshooting or missing of target, but quick to correct		Accurately directs instruments in the correct plane to target
Bimanual dexterity
1	2	3	4	5
Uses only one robotic arm, poor coordination		Uses both arms, but does not optimize interactions between arms		Expertly uses both arms in a complementary way to provide best exposure
Efficiency
1	2	3	4	5
Inefficient efforts, many uncertain movement, constantly changing focus or persisting without progress		Slow, but planned movements are reasonably organized		Confident, efficient and safe conduct, maintains focus on task, fluid progression
Tissue handling
1	2	3	4	5
Rough movements, injures tissues, poor grasper control, lack of tension		Handles tissues reasonably well, minor trauma to tissues, not optimal tension		Handles tissues well, applies appropriate tension, negligible injury to tissue
Autonomy
1	2	3	4	5
Unable to complete entire task, even with verbal guidance		Able to complete task safety with moderate guidance		Able to complete task independently without prompting
Endoscope control
1	2	3	4	5
Consistently does not optimize view and scope position, extremely frequent and inefficient adjustment		View and scope position is sometimes not optimal, needs to adjust scope position frequently		Controls view and scope position optimally and independently

Study design

Five procedures were performed by three operators (Novice-1, Novice-2 and Expert) ([Fig. 1]). Novice operators were overseas gastroenterologists who had no MASTER experience, but had performed conventional ESD, with Novice-1 having accumulated experience with approximately 500 ESD cases and Novice-2, more than 1000 such cases. Expert had previously performed eight animal and human MASTER ESD. To begin with, novice operators watched a video on the performance of a previous MASTER ESD and then underwent 15 minutes of practice in manipulating the robotic arms under virtual reality simulation, as well as in a dry laboratory setting. Then, each novice operator performed one ex vivo MASTER ESD on the rectum. Subsequently, each novice operator performed one in vivo MASTER ESD (a gastric ESD by Novice-1, and a sigmoid colon ESD by Novice-2). Expert separately performed an in vivo MASTER ESD on the sigmoid colon. Every procedure was self-evaluated using GEARS-E by all three participants, and by an independent evaluator, who had previously performed three MASTER ESD but was not involved in the procedures in this study. Statistically significant differences were evaluated by paired two-sided t-test followed by multiple testing correction using Holm method (P < alpha = 0.05 /n).

Fig. 1 Study design of in vivo and ex vivo animal experiments. Five procedures in all were performed by three operators (Novice-1, Novice-2 and Expert) and assessed by four evaluators (including self-evaluations) using GEARS-E.

Results

A total of five procedures were successfully completed, of which two were done each by Novice-1 and Novice-2, and one by Expert. Every procedure was assessed by the four evaluators including the Novice-1, Novice-2, Expert and the independent evaluator. The proficiency and performances of the operators as reflected by their mean scores from 4 evaluators in the 5 domains is summarized in [Table 2]. The mean global rating scores for Novice-1 first attempt, Novice-1 second attempt, Novice-2 first attempt, Novice-2 second attempt and Expert’s case were 13.0 (6 – 21), 16.0 (12 – 19), 13.3 (6 – 23), 15.5 (13 – 20) and 21.5 (14 – 25), respectively. The mean scores for each of the five domains were statistically higher in the second attempts compared to the first attempts for both Novice-1 (P = 0.0007 < alpha/4) and Novice-2 (P = 0.0486 < alpha/1) ([Fig. 2]). The results suggested that incremental changes in proficiency following repeated procedure. On the other hand, the mean scores for each of the five domains for Expert’s case were consistently higher than those for the two novice operators in both the first and second attempts (P = 0.0012 < alpha/2, P = 0.0008 < alpha/3, respectively).

Table 2 Means and ranges of scores in the five domains and global rating scores.

GEARS-E	Novice-1 1st	Novice-1 2nd	Novice-2 1st	Novice-2 2nd	Expert
Depth perception	2.5 (1 – 5)	3 (2 – 4)	2.5 (1 – 5)	3.3 (3 – 4)	4.3 (3 – 5)
Bimanual dexterity	2.8 (1 – 4)	3.3 (3 – 4)	2.5 (1 – 4)	3.3 (3 – 4)	4.3 (3 – 5)
Efficiency	2.8 (1 – 4)	3.5 (3 – 4)	3 (1 – 5)	3.3 (3 – 4)	4.3 (3 – 5)
Tissue handling	2.5 (1 – 4)	3.3 (2 – 4)	2.8 (1 – 5)	2.8 (2 – 4)	4.3 (2 – 5)
Autonomy	2.5 (2 – 4)	3 (2 – 4)	2.5 (1 – 4)	3 (2 – 4)	4.5 (3 – 5)
Global rating score	13 (6 – 21)	16 (12 – 19)	13.3 (6 – 23)	15.5 (13 – 20)	21.5 (14 – 25)

Fig. 2 Mean scores for each of five domains were statistically higher in the second attempts compared to the first attempts for both Novice-1 (P = 0.0007 < alpha/4) and Novice-2 (P = 0.0486 < alpha/1). Mean scores for each of five domains for Expert’s case were higher than those for the two novice operators in both the first and second attempts (P = 0.0012 < alpha/2, P = 0.0008 < alpha/3, respectively).

Discussion

While operation time, completeness of resection and adverse event rates are thought to be good indices for objective assessment of ESD procedures [6] [10] [11] [12], in reality, using them to compare operator skill or the effects of training across various clinical conditions can be challenging. For tackling this, a number of assessment tools have been generated [13] [14]. Because MASTER is a novel and unique endosurgical platform, an objective and reliable method of measuring training efficacy and outcome is needed.

In the current study, the means of the global rating scores and of each domain assessed increased alongside the increases in surgical experiences of MASTER. Especially when we put all domains together, statistically significant differences in parallel with experiences were observed. We deduced, therefore, that there is substantial possibility for GEARS-E to be an objective and consistent scale for assessment of proficiency in performing MASTER procedure.

On the other hand, we noted the wide-ranging scores of individual evaluators. For standardization, the anchoring method we applied probably didn’t work well. During evaluator training, the evaluators in the GOALS study were encouraged to use the full range of scores [8]. Another possible mean to help evaluators in standardizing their assessment of skills based on different skill levels is to let them watch video recordings of typical procedures performed by novice, intermediate and expert operators.

This study had some limitations. First, the evaluators were not blinded, and on the contrary, we included self-evaluations besides evaluations by other independent evaluators. Second, because this was a pilot study, sample size was small. Furthermore, due to the nature of ESD, it is challenging to equalize difficulty factors such as lesion location and endoscopic conditions. Finally, to focus on novice training, the sixth domain was excluded from the assessment.

Conclusion

In conclusion, results using GEARS-E showed correlations with surgical experience in MASTER ESD. As an assessment tool for evaluation of surgical skills, GEARS-E has a great potential for application in MASTER procedures. For the next step, we will design a training program that integrates objective GEARS-E skill assessment so that we can promote MASTER safely and assess its effectiveness across centers around the world.

Competing interests

Dr. Chiu served on the scientific advisory board of EndoMaster Pte Ltd. Drs. Phee and Ho are cofounders of EndoMaster Pte Ltd.

• References

• 1 Ono H, Yao K, Fujishiro M. et al. Guidelines for endoscopic submucosal dissection and endoscopic mucosal resection for early gastric cancer. Dig Endosc 2016; 28: 3-15
• 2 Akutsu Y, Uesato M, Shuto K. et al. The overall prevalence of metastasis in T1 esophageal squamous cell carcinoma: a retrospective analysis of 295 patients. Ann Surg 2013; 257: 1032-1038
• 3 Tanaka S, Kashida H, Saito Y. et al. JGES guidelines for colorectal endoscopic submucosal dissection/endoscopic mucosal resection. Dig Endosc 2015; 27: 417-434
• 4 Phee SJ, Low SC, Huynh VA. et al. Master and slave transluminal endoscopic robot (MASTER) for natural orifice transluminal endoscopic surgery (NOTES). Conf Proc IEEE Eng Med Biol Soc 2009; 2009: 1192-1195
• 5 Takeshita N, Ho KY, Phee SJ. et al. Feasibility of performing esophageal endoscopic submucosal dissection using master and slave transluminal endoscopic robot. Endoscopy 2017; 49: E27-28
• 6 Phee SJ, Reddy N, Chiu PW. et al. Robot-assisted endoscopic submucosal dissection is effective in treating patients with early-stage gastric neoplasia. Clin Gastroenterol Hepatol 2012; 10: 1117-1121
• 7 Ho KY, Phee SJ, Shabbir A. et al. Endoscopic submucosal dissection of gastric lesions by using a Master and Slave Transluminal Endoscopic Robot (MASTER). Gastrointest Endosc 2010; 72: 593-599
• 8 Vassiliou MC, Feldman LS, Andrew CG. et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 2005; 190: 107-113
• 9 Aghazadeh MA, Jayaratna IS, Hung AJ. et al. External validation of Global Evaluative Assessment of Robotic Skills (GEARS). Surg Endosc 2015; 29: 3261-3266
• 10 Tekkis PP, Senagore AJ, Delaney CP. et al. Evaluation of the learning curve in laparoscopic colorectal surgery: comparison of right-sided and left-sided resections. Ann Surg 2005; 242: 83-91
• 11 Yamamoto S, Uedo N, Ishihara R. et al. Endoscopic submucosal dissection for early gastric cancer performed by supervised residents: assessment of feasibility and learning curve. Endoscopy 2009; 41: 923-928
• 12 Chiu PW, Phee SJ, Bhandari P. et al. Enhancing proficiency in performing endoscopic submucosal dissection (ESD) by using a prototype robotic endoscope. Endosc Int Open 2015; 5: E439-E442
• 13 Goff BA, Lentz GM, Lee D. et al. Development of an objective structured assessment of technical skills for obstetric and gynecology residents. Obstet Gynecol 2000; 96: 146-150
• 14 Vassiliou MC, Kaneva PA, Poulose BK. et al. Global Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy. Surg Endosc 2010; 24: 1834-1841