Real-Time Image Overlay for Decision Support in Endoscopic Mitral Valve Surgery

Objectives: Endoscopic mitral valve repair is the gold standard therapy for isolated mitral valve repair in specialized centers. Therapy planning and intraoperative imaging are crucial because the field of view is limited. To support therapy planning and enable live intraoperative image overlay (i.e., pre-op echocardiographic data into the endoscopic images), modeling algorithms have been invented. Here, we propose a real-time-capable deep-learning–based approach to detect and segment the relevant anatomical structures and instruments as proof-of-concept for landmark matching.

Methods: Stereo endoscopic image datasets from minimally invasive mitral valve procedures were used, resulting in 540 frames. Medical experts annotated relevant anatomical structures in these images. Eight sets were used to train the model, one was used for validation. Three deep learning algorithms were applied: U-Net, Google's DeepLab, and Obelisk-Net. For training, 8-fold cross validation was used. Different forms of data augmentation were used and transfer learning approaches were tested. To reduce squishing effect and speed up model training, images were resized to a 960 × 512 resolution from the initial 1,920 × 1,080 resolution.

Result: All algorithms were able to detect the structures of interest. Based on the Dice-Score, U-Net, and DeepLab architectures performed almost equally well with a mean score of 0.93 with required times for segmentation interference of 120 and 90 ms, respectively. The Obelisk-Net failed to achieve a mean Dice-Score of 0.6 or more but required less time (20 ms). The Obelisk-Net does not perform well with respect to the contour distance either. Comparing the U-Net and DeepLab distance distributions, the latter architecture is slightly favored, with a lower mean contour distance, and a smaller mean for the Hausdorff distance measurement. Furthermore, the distribution of the DeepLab is narrower compared with the U-Net, which indicates it is not only performing better on average but also in worst-case scenarios.

Conclusion: DeepLab model showed superiority in regard to predicting and detecting anatomical structures and instruments during endoscopic mitral valve repair. U-Net was inferior, and Obelisk-Net failed to achieve adequate performance. Deep-Learning algorithms in combination with medical expertise are powerful tools to enrich modern minimally invasive surgery by adding models and predictions as overlays in the live endoscopic images. These features can be used as decision support, for training purposes, and others.

Publication History

Article published online:
19 February 2021