Endosc Int Open 2018; 06(08): E1065-E1069
DOI: 10.1055/a-0640-3123
Original article
© Georg Thieme Verlag KG Stuttgart · New York

Global Evaluative Assessment of Robotic Skills in Endoscopy (GEARS-E): objective assessment tool for master and slave transluminal endoscopic robot

Nobuyoshi Takeshita
1  Department of Medicine, National University of Singapore, Singapore
,
Soo Jay Phee
2  School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore
,
Philip WaiYan Chiu
3  Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China
,
Khek Yu Ho
4  National University Hospital, Singapore
› Author Affiliations
Further Information

Corresponding author

Nobuyoshi Takeshita, MD, PhD
Department of Medicine
National University of Singapore
1E Kent Ridge Road
NUHS Tower Block, Level 10
University Medicine Cluster
Singapore 119260   
Fax: +-65-6772-4361   

Publication History

submitted 13 December 2017

accepted after revision 30 April 2018

Publication Date:
10 August 2018 (online)

 

Abstract

Background and study aims The Master and Slave Transluminal Endoscopic Robot (MASTER) is a novel robotic endosurgical system with two operating arms that offer multiple degrees of freedom. We developed a new assessment tool, the Global Evaluative Assessment of Robotic Skills in Endoscopy (GEARS-E), derived from existing tools in laparoscopic and robotic surgery, and evaluated its utility in the performance of procedures using MASTER.

Methods This was a pilot study conducted in vivo and ex vivo on animals. Three operators (Novice-1, Novice-2 and Expert) performed a total of five endoscopic submucosal dissections (ESD) using MASTER. Novice operators had no MASTER experience and the expert had previously performed eight MASTER ESDs. Operator performance was assessed by four independent evaluators using GEARS-E, which has a maximum score of 25 for five domains representing various skill-related variables (depth perception, bimanual dexterity, efficiency, tissue handling and autonomy).

Results The mean global rating scores for Novice-1 first attempt, Novice-1 second attempt, Novice-2 first attempt, Novice-2 second attempt and Expertʼs cases were 13.0, 16.0, 13.3, 15.5, and 21.5, respectively. The mean scores of each of the five domains were statistically higher for the second attempts compared to the first attempts for both Novice-1 and Novice-2. The mean scores of each of the five domains for the Expertʼs case were consistently higher than those for the two novice operators in both their first and second attempts.

Conclusion Results using GEARS-E showed correlations between surgical experience and MASTER ESD. As an assessment tool for evaluation of surgical skills, GEARS-E has great potential for application in MASTER procedures.


#

Introduction

Endoscopic submucosal dissection (ESD) has become standard care for early gastrointestinal cancers [1] [2] [3]. Master and Slave Transluminal Endoscopic Robot (MASTER) is a novel robotic endosurgical system that has two operating arms equipped with multiple degrees of freedom, enabling complex maneuvers [4]. The feasibility of performing MASTER ESD has been shown in previous studies [5] [6] [7]. Going forward, the value of MASTER will need to be evaluated in terms of the user’s learning curve and training efficacy. Therefore, it is essential to develop an appropriate training program and a suitable skill assessment tool.

To provide an objective assessment tool in minimally invasive surgery, the Global Operative Assessment of Laparoscopic Skills (GOALS) was developed [8]. More recently, the Global Evaluative Assessment of Robotic Skills (GEARS), which was modeled after GOALS, was also developed [9]. There has not been an objective assessment tool developed for endoluminal surgical platforms such as MASTER. In this study, we explored use of a new scoring method, the Global Evaluative Assessment of Robotic Skills in Endoscopy (GEARS-E), which was modified from both GOALS and GEARS in evaluation of the learning curve of novice operators using MASTER.


#

Methods

Procedures

This study involved in vivo and ex vivo experiments. Participating operators performed ESD using MASTER on two adult female pigs, as well as on isolated pig tissues obtained from a commercial source. All experiments were conducted at Innoheart Pte Ltd with prior approval from the Institutional Animal Care and Use Committee. As for the in vivo experiments, the endoscope was introduced under general anesthesia and the operator telemanipulated the two robotic arms, one holding a cautery knife and the other a grasper, via cables that passed through the two channels of the endoscope. For the ex vivo experiments, the isolated colorectal segment was mounted on a stand to simulate its normal orientation. Then, an imaginary lesion was marked on the mucosa and MASTER ESD was performed. The pigs were euthanized after the procedures.


#

GEARS-E

In creating GEARS-E, we expanded the domains in GOALS and GEARS to encompass features unique to MASTER procedures ([Table 1]). Modeled after GOALS and GEARS, GEARS-E comprises six domains representing various skill-related variables (depth perception, bimanual dexterity, efficiency, tissue handling, autonomy and endoscope control). Assessment of each domain is done by scoring on a 5-point Likert scale with performance anchors at 1, 3 and 5. A rating of 1 denotes the lowest proficiency while 5 is the highest. As the focus of this study was on the novice’s training on robotic manipulators, we left the endoscope manipulation to a skilled endoscopist independently. To minimize bias from this, we excluded the sixth domain, endoscopic control, from the assessment. Therefore, the global rating score for this study was the sum of the scores for each of the remaining five domains, the maximum total score of which is 25.

Table 1

Six domains of GEARS-E.

Depth perception

1

2

3

4

5

Constantly overshoots target, wide swings, slow to correct

Some overshooting or missing of target, but quick to correct

Accurately directs instruments in the correct plane to target

Bimanual dexterity

1

2

3

4

5

Uses only one robotic arm, poor coordination

Uses both arms, but does not optimize interactions between arms

Expertly uses both arms in a complementary way to provide best exposure

Efficiency

1

2

3

4

5

Inefficient efforts, many uncertain movement, constantly changing focus or persisting without progress

Slow, but planned movements are reasonably organized

Confident, efficient and safe conduct, maintains focus on task, fluid progression

Tissue handling

1

2

3

4

5

Rough movements, injures tissues, poor grasper control, lack of tension

Handles tissues reasonably well, minor trauma to tissues, not optimal tension

Handles tissues well, applies appropriate tension, negligible injury to tissue

Autonomy

1

2

3

4

5

Unable to complete entire task, even with verbal guidance

Able to complete task safety with moderate guidance

Able to complete task independently without prompting

Endoscope control

1

2

3

4

5

Consistently does not optimize view and scope position, extremely frequent and inefficient adjustment

View and scope position is sometimes not optimal, needs to adjust scope position frequently

Controls view and scope position optimally and independently


#

Study design

Five procedures were performed by three operators (Novice-1, Novice-2 and Expert) ([Fig. 1]). Novice operators were overseas gastroenterologists who had no MASTER experience, but had performed conventional ESD, with Novice-1 having accumulated experience with approximately 500 ESD cases and Novice-2, more than 1000 such cases. Expert had previously performed eight animal and human MASTER ESD. To begin with, novice operators watched a video on the performance of a previous MASTER ESD and then underwent 15 minutes of practice in manipulating the robotic arms under virtual reality simulation, as well as in a dry laboratory setting. Then, each novice operator performed one ex vivo MASTER ESD on the rectum. Subsequently, each novice operator performed one in vivo MASTER ESD (a gastric ESD by Novice-1, and a sigmoid colon ESD by Novice-2). Expert separately performed an in vivo MASTER ESD on the sigmoid colon. Every procedure was self-evaluated using GEARS-E by all three participants, and by an independent evaluator, who had previously performed three MASTER ESD but was not involved in the procedures in this study. Statistically significant differences were evaluated by paired two-sided t-test followed by multiple testing correction using Holm method (P < alpha = 0.05 /n).

Zoom Image
Fig. 1 Study design of in vivo and ex vivo animal experiments. Five procedures in all were performed by three operators (Novice-1, Novice-2 and Expert) and assessed by four evaluators (including self-evaluations) using GEARS-E.

#
#

Results

A total of five procedures were successfully completed, of which two were done each by Novice-1 and Novice-2, and one by Expert. Every procedure was assessed by the four evaluators including the Novice-1, Novice-2, Expert and the independent evaluator. The proficiency and performances of the operators as reflected by their mean scores from 4 evaluators in the 5 domains is summarized in [Table 2]. The mean global rating scores for Novice-1 first attempt, Novice-1 second attempt, Novice-2 first attempt, Novice-2 second attempt and Expert’s case were 13.0 (6 – 21), 16.0 (12 – 19), 13.3 (6 – 23), 15.5 (13 – 20) and 21.5 (14 – 25), respectively. The mean scores for each of the five domains were statistically higher in the second attempts compared to the first attempts for both Novice-1 (P = 0.0007 < alpha/4) and Novice-2 (P = 0.0486 < alpha/1) ([Fig. 2]). The results suggested that incremental changes in proficiency following repeated procedure. On the other hand, the mean scores for each of the five domains for Expert’s case were consistently higher than those for the two novice operators in both the first and second attempts (P = 0.0012 < alpha/2, P = 0.0008 < alpha/3, respectively).

Table 2

Means and ranges of scores in the five domains and global rating scores.

GEARS-E

Novice-1 1st

Novice-1 2nd

Novice-2 1st

Novice-2 2nd

Expert

Depth perception

 2.5 (1 – 5)

 3 (2 – 4)

 2.5 (1 – 5)

 3.3 (3 – 4)

 4.3 (3 – 5)

Bimanual dexterity

 2.8 (1 – 4)

 3.3 (3 – 4)

 2.5 (1 – 4)

 3.3 (3 – 4)

 4.3 (3 – 5)

Efficiency

 2.8 (1 – 4)

 3.5 (3 – 4)

 3 (1 – 5)

 3.3 (3 – 4)

 4.3 (3 – 5)

Tissue handling

 2.5 (1 – 4)

 3.3 (2 – 4)

 2.8 (1 – 5)

 2.8 (2 – 4)

 4.3 (2 – 5)

Autonomy

 2.5 (2 – 4)

 3 (2 – 4)

 2.5 (1 – 4)

 3 (2 – 4)

 4.5 (3 – 5)

Global rating score

13 (6 – 21)

16 (12 – 19)

13.3 (6 – 23)

15.5 (13 – 20)

21.5 (14 – 25)

Zoom Image
Fig. 2 Mean scores for each of five domains were statistically higher in the second attempts compared to the first attempts for both Novice-1 (P = 0.0007 < alpha/4) and Novice-2 (P = 0.0486 < alpha/1). Mean scores for each of five domains for Expert’s case were higher than those for the two novice operators in both the first and second attempts (P = 0.0012 < alpha/2, P = 0.0008 < alpha/3, respectively).

#

Discussion

While operation time, completeness of resection and adverse event rates are thought to be good indices for objective assessment of ESD procedures [6] [10] [11] [12], in reality, using them to compare operator skill or the effects of training across various clinical conditions can be challenging. For tackling this, a number of assessment tools have been generated [13] [14]. Because MASTER is a novel and unique endosurgical platform, an objective and reliable method of measuring training efficacy and outcome is needed.

In the current study, the means of the global rating scores and of each domain assessed increased alongside the increases in surgical experiences of MASTER. Especially when we put all domains together, statistically significant differences in parallel with experiences were observed. We deduced, therefore, that there is substantial possibility for GEARS-E to be an objective and consistent scale for assessment of proficiency in performing MASTER procedure.

On the other hand, we noted the wide-ranging scores of individual evaluators. For standardization, the anchoring method we applied probably didn’t work well. During evaluator training, the evaluators in the GOALS study were encouraged to use the full range of scores [8]. Another possible mean to help evaluators in standardizing their assessment of skills based on different skill levels is to let them watch video recordings of typical procedures performed by novice, intermediate and expert operators.

This study had some limitations. First, the evaluators were not blinded, and on the contrary, we included self-evaluations besides evaluations by other independent evaluators. Second, because this was a pilot study, sample size was small. Furthermore, due to the nature of ESD, it is challenging to equalize difficulty factors such as lesion location and endoscopic conditions. Finally, to focus on novice training, the sixth domain was excluded from the assessment.


#

Conclusion

In conclusion, results using GEARS-E showed correlations with surgical experience in MASTER ESD. As an assessment tool for evaluation of surgical skills, GEARS-E has a great potential for application in MASTER procedures. For the next step, we will design a training program that integrates objective GEARS-E skill assessment so that we can promote MASTER safely and assess its effectiveness across centers around the world.


#
#

Competing interests

Dr. Chiu served on the scientific advisory board of EndoMaster Pte Ltd. Drs. Phee and Ho are cofounders of EndoMaster Pte Ltd.


Corresponding author

Nobuyoshi Takeshita, MD, PhD
Department of Medicine
National University of Singapore
1E Kent Ridge Road
NUHS Tower Block, Level 10
University Medicine Cluster
Singapore 119260   
Fax: +-65-6772-4361   


Zoom Image
Fig. 1 Study design of in vivo and ex vivo animal experiments. Five procedures in all were performed by three operators (Novice-1, Novice-2 and Expert) and assessed by four evaluators (including self-evaluations) using GEARS-E.
Zoom Image
Fig. 2 Mean scores for each of five domains were statistically higher in the second attempts compared to the first attempts for both Novice-1 (P = 0.0007 < alpha/4) and Novice-2 (P = 0.0486 < alpha/1). Mean scores for each of five domains for Expert’s case were higher than those for the two novice operators in both the first and second attempts (P = 0.0012 < alpha/2, P = 0.0008 < alpha/3, respectively).