Acceptance and feasibility of an augmented reality-based navigation system with optical tracking for percutaneous procedures in interventional radiology – a simulation-based phantom study

Abstract

Purpose

Augmented reality (AR) projects additional information into the user’s field of view during interventions. The aim was to evaluate the acceptance and clinical feasibility of an AR system and to compare users with different levels of experience. A system was examined that projects a CT-generated 3D model of a phantom into the field of view using a HoloLens 2, whereby the tracked needle is displayed and navigated live. A projected ultrasound image is used for live control of the needle positioning. This should minimize radiation exposure and improve orientation.

Materials and Methods

The acceptance and usability of the AR navigation system was evaluated by 10 physicians and medical students with different levels of experience by performing punctures with the system in a phantom. The required time was then compared and a questionnaire was completed to assess clinical acceptance and feasibility. For statistical analysis, frequencies for qualitative characteristics, location and dispersion measures for quantitative characteristics and Spearman rank correlations for correlations were calculated.

Results

9 out of 10 subjects hit all 5 target regions in the first attempt, taking an average of 29:39 minutes for all punctures. There was a significant correlation between previous experience in interventional radiology, years in the profession, and the time required. Overall, the time varied from an average of 43:00 min. for medical students to 15:00 min. for chief physicians. All test subjects showed high acceptance of the system and rated especially the potential clinical feasibility, the simplification of the puncture, and the image quality positively. However, the majority require further training for sufficient safety in use.

Conclusion

The system offers distinct advantages for navigation and orientation, facilitates percutaneous interventions during training and enables professionally experienced physicians to achieve short intervention times. In addition, the system improves ergonomics during the procedure by making important information always directly available in the field of view and has the potential to reduce the radiation exposure of staff in particular by combining AR and sonography and thus shortening CT-fluoroscopy times.

Key Points

• AR navigation offers advantages for orientation during percutaneous radiological interventions.

• The subjects would like to use the AR system in everyday clinical practice on patients.

• AR improves ergonomics by making important information directly available in the field of view.

• The combination of AR and sonography can significantly reduce radiation exposure for staff.

Citation Format

• Rohmer K, Becker M, Georgiades M et al. Acceptance and feasibility of an augmented reality-based navigation system with optical tracking for percutaneous procedures in interventional radiology - a simulation-based phantom study. Rofo 2025; 197: 936–944

 

MissingLinkText

Keywords

interventional procedures - biopsy - ultrasound - interventional radiology - augmented reality

Introduction

Augmented reality (AR), which allows additional information to be overlaid on top of reality, has many benefits beyond interventional radiology. For example, additional information such as CT or MRI images can be made available directly in the field of view during the intervention. This includes three-dimensional holograms of the patient’s organs or 3D navigation data of a puncturing needle. Head-mounted displays (HMDs) are widely used to display such AR images. One such HMD is the Microsoft HoloLens 2. The potential use of AR in medicine has already been demonstrated in other studies for many specialties. These include general surgery [1], orthopedics [2], thyroid surgery [3], urology [4], and vascular surgery [5].

CT fluoroscopy, which is currently used for complex punctures, is a challenging procedure that requires a high degree of spatial awareness on the part of the interventionalist. The position of the needle in the three-dimensional body must be abstracted from two-dimensional axial CT images and can only be supported to a limited extent by multiplanar reconstructions. Augmented reality could be a suitable method to simplify this orientation. By displaying 3D projections and the associated depth perception, better transferability to the patient's body can be achieved [5]. This simplifies puncture with alternative access routes, which could minimize the risk of injury to critical structures and thus increase patient safety.

Studies in the field of urology have already shown that augmented reality can significantly minimize procedure time and achieve better quality results [4]. Shorter procedure times could help to compensate for the increasing workload with punctures.

AR has many additional benefits. For example, data (3D models, live images, etc.) can be projected directly into the user’s field of view. For example, it is possible to combine fusion images from CT and other modalities with 3D holograms of the patient’s organs. In preparation for biopsies, it offers the possibility of planning puncture paths on three-dimensional models and displaying them during the procedure. Combined with live tracking of needles and other instruments, a new and cost-effective navigation system for minimally invasive interventions can be realized. Importantly, the information does not need to be displayed on an additional screen but can be projected directly onto the HMD in the user’s field of view. This leads to improved ergonomics during the procedure and a focus on the essential content [6]. By eliminating the need for additional screens, which often have to be positioned in inconvenient locations, associated problems such as back, shoulder, and neck pain can be avoided [7].

It also minimizes the otherwise increased risk of iatrogenic injury due to a disrupted visual-motor axis [8].

An HMD can be worn under sterile conditions [3]. This offers the possibility of using the navigation system during a procedure, potentially reducing the frequency and duration of CT fluoroscopy and thus radiation exposure, especially for medical staff.

The primary objective of this study was to determine the acceptance and clinical feasibility of a HoloLens 2-based AR system for minimally invasive CT-guided interventional radiology procedures. The secondary objective was to evaluate the learning curves of subjects with different levels of experience.

 

MissingLinkText

Material and methods

Hardware

An overview of the material used can be seen in [Fig. 1].

Fig. 1 Equipment.

A HoloLens 2 (version: 20348.1542, Microsoft Corporation, Redmond, USA) was used as the AR system. This can be controlled entirely by hand gestures to allow operation under sterile conditions. The HoloLens 2 displays have a 2K resolution in a 3:2 format, which corresponds to a resolution of >2500 light points per radian [9]. The HMD was connected to the workstation via a 5 GHz Wi-Fi network.

A stereo camera (MicronTracker 3 Hx40, ClaroNav Inc., Toronto, Canada) was used for optical tracking.

The punctures were performed on a phantom (CIRS triple modality 3D abdominal phantom, Model 057A, Sun Nuclear Corporation, Melbourne, USA) with internal structures (ribs, spine, kidneys, liver, hepatic vein, lungs), which allows both ultrasound and CT imaging ([Fig. 2]). Needles with a working length of 150 mm (17G, 1.4 × 180 mm, KLS Martin SE & Co. KG, Tuttlingen, Germany) were used for the puncture simulation.

Fig. 2 CT scan of phantom.

For optical tracking, six optical trackers with a small metal ball in the optical center were attached to the phantom. This ball can be detected on CT, so that the CT data can later be fused with the optical images from the camera. Special optical markers (Holo4Med S.A., Białystok, Poland) were also used to track the needle. A curved array probe (3–11 MHz) (S40, SonoScape Medical Corp., Shenzhen, China) or a C5–2 probe (2–5 MHz) (ACUSON Freestyle, Siemens Healthineers AG, Forchheim, Germany) was used for sonography.

 

MissingLinkText

Software and application:

The software that was used was an application called HoloMIAI (Holo4Med S.A., Białystok, Poland). The DICOM files of the phantom were converted by the software into a 3D model with segmented internal structures. In this model, needle paths can be planned by defining entry and target points. The HoloLens 2 displays the 3D model of the phantom, including the planned needle trajectories, to the subject. A line is displayed between the planned entry point and the target point, which extends out of the phantom for easy orientation. During the puncture, the puncture needle is also projected onto the 3D model and must overlap with the extended puncture line of the 3D model in order to puncture the target structure correctly. In addition, the target point changes color from red to green as soon as the extended needle tip is pointed at it ([Fig. 3] and [Fig. 4]). Furthermore, the subjects have another tool at their disposal called “Aim-Panel”. This is an aiming guide consisting of a red ring, a blue ring, and a white dot. For a correct puncture, both rings must be placed over the white dot. The blue ring indicates the distance of the needle tip to the entry point and the red ring indicates the correct alignment of the needle tip to the target. In addition, the distance to the target structure is indicated by a bar that fills as the target is approached ([Fig. 3] and [Fig. 4]). At a distance of approximately 2 cm from the target, ultrasound should also be used. This is projected into the user’s field of view in the HoloLens 2 as well and serves as a live imaging modality during the final puncture ([Fig. 4]). All three components (3D model, Aim-Panel, and ultrasound image) can be freely positioned in the room by the user using hand gestures and can be shown or hidden as desired.

Fig. 3 AR image of the HoloLens with the needle correctly aimed at the target structure.

Fig. 4 AR image during the final puncture with ultrasound.

 

MissingLinkText

Study program

The acceptance and feasibility of the AR system was evaluated in subjects with different levels of experience and age (3 medical students, 2 residents, 3 specialists, 1 senior physician, and 1 chief physician). In addition, the subjects’ professional experience was recorded in the form of years in radiology and they were asked to state the number of CT fluoroscopies, ultrasound examinations, and ultrasound-assisted punctures they had performed. Previous experience with AR or VR (virtual reality) was also documented. All subjects first received an introduction to the AR system and underwent the general tutorial of the HoloLens 2 from Microsoft, in which they learned about the general operation. The HoloLens was also calibrated to the subject’s eyes. The subjects were then given the opportunity to familiarize themselves with the software. They were first given an introduction to the various tools and options of the HoloMIAI system and how to use them. They were also given an introduction to the main voice commands. These were also available as a list during the tests. After the subjects had familiarized themselves with the operation of the system, a practice phase with a total of 3 punctures followed: a first practice puncture under standardized instructions in a target region 6 cm deep and approximately 8 mm in diameter, followed by two further punctures with different needle paths and target sizes for training purposes ([Fig. 5]).

Fig. 5 User during puncture.

After the training phase, the test subjects were asked to puncture five different target lesions of different sizes and with different lengths and angulations of the needle path, avoiding the critical structures of the model (lung and hepatic vein). The diameters of the lesions varied between 7 mm and 15 mm and the length of the needle path between 65 mm and 143 mm.

The needle tip positions were then documented and evaluated using a CT scan. The time required from needle insertion for the first puncture until the subject declared the completion of the fifth lesion puncture was documented. If not all target lesions were hit correctly, a new round of 5 punctures was performed. Any problems and their reasons were also documented.

After the puncture, the subjects were asked to complete a questionnaire on the clinical feasibility, safety, and handling of the system. The subjects were asked to rate questions ([Table 1]) on a Likert scale from ‘strongly agree’ (5) to ‘strongly disagree’ (1).

Table 1 Questionnaire regarding clinical applicability, safety, and handling with results.

Questions	N	Mean	Standard deviation	Median	Minimum	Maximum
I can imagine using HoloLens in everyday clinical practice.	10	4.60	.516	5.00	4	5
The system simplified orientation during the intervention.	10	4.30	.949	5.00	3	5
The system simplified needle navigation.	10	4.70	.675	5.00	3	5
I quickly got used to using the system.	10	3.80	.919	4.00	2	5
Operation of the HoloLens was user-friendly.	10	3.90	.738	4.00	3	5
I found the display of information in the field of vision useful.	10	4.40	.516	4.00	4	5
The image quality of the HoloLens was sufficient for the intended purpose.	10	4.70	.483	5.00	4	5
Puncture outside the CT gantry simplified the procedure. (physicians only)	7	4.57	.787	5.00	3	5
While using the system I felt secure.	10	3.60	1.075	4.00	2	5
Augmented reality should play a role in interventional radiology in the future.	10	4.50	.707	5.00	3	5
Augmented reality should play a role in the training of young physicians.	10	4.90	.316	5.00	4	5
I feel confident using the system.	10	3.20	1.033	3.00	2	5
I would like to use the system for percutaneous procedures on patients.	10	4.40	.699	4.50	3	5
I still need more training with the system.	10	4.20	1.317	5.00	1	5

 

MissingLinkText

Statistical analysis

IBM SPSS Statistics 28.0 (IBM Corp., Armonk, NY, USA) was used for statistical analysis. First, a descriptive survey of the recorded characteristics was conducted. For qualitative characteristics, frequencies were calculated and presented as absolute numbers and percentages. For quantitative characteristics, measures of location and dispersion were determined. Spearman’s rank correlations were used to examine correlations between quantitative variables. These were interpreted according to Cohen (1988): |rs| = 0.10 – weak correlation, |rs| = 0.30 – moderate correlation, |rs| = 0.50 – strong correlation).

All tests were two-sided, and a p-value ≤ 0.05 was considered statistically significant.

 

MissingLinkText

 

MissingLinkText

Results

9 out of 10 subjects were able to hit all 5 target structures on the first attempt. Only one resident needed a second round. On average, the subjects needed 29:39 minutes for 5 successful punctures. It was not possible to examine the distance to the lesion center, as the measurement inaccuracy would be too high for target lesions measuring only 3.5 mm in radius, even with 1 mm CT slices because of 1.4 mm thick needles and significant metal artifacts. Therefore, only the classification as “hit” or “no hit”, which is relevant in practice, was used. There was a significant correlation between the number of years in radiology and the time required for the punctures (r_s=–0.787; p=0.007), as well as between the time required and previous experience with CT fluoroscopy (r_s=–0.755; p=0.012), sonography (r_s=–0.632; p=0.050) and sonography-assisted punctures (r_s=–0.745; p=0.013). Three of the subjects reported that they had once experienced VR or AR outside of a medical context. However, this had no detectable connection with a faster puncture time.

Medical students took an average of 43:00 min, residents 34:30 min, specialists 22:10 min, senior physicians 17:00 min, and chief physicians 15:00 min ([Fig. 6] and [Table 2]).

Fig. 6 Graphical representation of the time required for 5 punctures over the experience level.

Table 2 Times required for 5 successful punctures.

Experience level	N	Mean	Standard deviation	Standard error of the mean	Median	Minimum	Maximum
Chief physician	1	15:00	.	.
Senior physician	1	17:00	.	.
Specialist	3	22:10	05:45	03:19	22:00	16:30	28:00
Resident	2	34:30	20:30	14:29	34:30	20:00	49:00
Medical student	3	43:00	02:00	01:09	43:00	41:00	45:00
Total	10	29:39	13:24	04:14	25:00	15:00	49:00

In the survey of participants, mainly positive and largely supportive statements were collected. As part of the questionnaire with a Likert scale of 1 to 5 points, the mean value (M) of the answers given was calculated. The test subjects stated that they could imagine using the HoloLens in everyday clinical practice (M=4.6), that the system simplified needle navigation (M=4.7), that the image quality was sufficient for the intended purpose (M=4.7), and that puncture outside the CT gantry simplified the procedure (M=4.57). The user-friendliness and quick familiarization with the handling were rated with an average of M=3.9 and M=3.8, respectively. The question of whether the subject felt safe during the puncture was rated with M=3.6. With an average rating of M=4.2, the test subjects agreed that they would like to use the system for percutaneous procedures on patients in the future. The need for further training was also rated at M=4.2 ([Table 1]). In general, there was no recognizable correlation between the scores given by medical students and more experienced test subjects. The only striking finding was that the chief physician surveyed already felt so confident with the system that he rated the need for further training as 1 and thus differed considerably from the rating of the other test subjects (M=4.2).

Overall, technical problems were rare. For two subjects, the HoloLens switched off due to overheating. This was presumably due to the additional computational load caused by the live-view of the HoloLens image to a PC during the study. There were two short transmission problems with the Wi-Fi, and one subject’s needle marker bent slightly, resulting in incorrect tracking of the needle.

 

MissingLinkText

Discussion

The results show that the AR system has many advantages for percutaneous procedures in interventional radiology, for example, improved orientation during the intervention, which is particularly advantageous for needle navigation. In addition, the integration of live ultrasound imaging while maintaining the possibility of CT fluoroscopy has the potential to drastically reduce radiation exposure for staff without reducing the safety of needle navigation in patients.

A significant correlation between professional experience and previous experience in interventional radiology of the test subjects and the time required for the punctures was to be expected and was confirmed here. Nevertheless, it should be emphasized that all medical students, without any professional experience, were also able to successfully perform the punctures with this system in the first attempt. This advantage in percutaneous procedures for completely inexperienced medical students is in line with results with other AR systems (without US) [10]. This shows the potential to flatten the learning curves for percutaneous procedures in interventional radiology and thus enable the training of more interventionalists through faster training.

User acceptance of the system was very high and handling was perceived as intuitive. In addition, all of the test subjects surveyed could well imagine using the system in everyday clinical practice. However, 80% of the test subjects stated that they needed further training with the system, which probably also explains the relatively lower rating regarding feeling safe during the puncture.

The applicability of 3D models displayed using augmented reality for punctures has been confirmed in other studies [10] [11] [12]. Studies have also already shown the applicability of ultrasound during CT-guided percutaneous procedures, with a significant reduction in radiation exposure [13].

However, this study is one of the first to evaluate a system that combines optically assisted needle tracking with AR projection of a 3D model and live ultrasound displayed in the user’s field of vision for percutaneous procedures. The superiority of a US image displayed using AR for percutaneous biopsies, particularly due to improved ergonomics [6], more comfortable working [14], and improved precision [15], has already been confirmed in other studies. The possibility of combining US and CT images for AR-assisted punctures [16], as well as instrument tracking and AR-projected ultrasound, has also already been demonstrated [17]. The applicability of a system comparable to the one tested here, with 3D navigation and ultrasound, has already been demonstrated for prostate punctures [18].

The average puncture time of 5:56 min/puncture with the AR/US procedure tested here in a phantom with the hepatic vein and lung as the main risk structures is comparable with the results of other studies with AR-based puncture systems. Here, the puncture times in a phantom without any risk structures with AR navigation in combination with CT fluoroscopy were 4:42 min/puncture [10] and 9:24 min/puncture for purely AR-supported punctures in a human cadaver [11]. It should be noted that the systems compared here differ significantly in terms of technology, but there are indications that the puncture time increases with the increase and complexity of risk structures.

One challenge is the usual deformation or bending of the needle in the phantom or, in the future, in the patient and the associated, not 100% straight puncture path, as the needle tip is abstracted from the marker at the end of the needle using a straight line as part of optical tracking. Therefore, continuous monitoring by live imaging modalities such as ultrasound is required, especially for the last 2–3 cm of the puncture path. It would also be conceivable to connect the system to bend-sensitive needles [19].

Another important limitation is that this is a stationary phantom that has no respiratory movement. This problem could be minimized by combining it with respiratory movement monitoring [20]. It is also conceivable that the artifacts of optical tracking caused by respiratory movement could be compensated by performing the final puncture under live imaging.

In patients or target structures where imaging by ultrasound is not possible, live imaging of the final end segment by CT fluoroscopy could also be considered [10].

In combination with ultrasound, the use of the system is limited to puncture sites where ultrasound is possible. It is, therefore, particularly suitable for punctures in the abdomen, for example for liver or kidney punctures. CT fluoroscopy should always be available if an adequate ultrasonic window cannot be found.

Based on these promising properties, we have already initiated a randomized clinical trial (prospective evaluation of an AR-based procedure for percutaneous procedures in interventional radiology).

In summary, the system has high potential for practical application, in particular the potential reduction of radiation exposure while maintaining safety through live imaging, as well as the possible improvement of ergonomics and orientation could possibly lead to a more efficient, user-friendly, and safer intervention.

 

MissingLinkText

Clinical relevance

• The AR system provides improved orientation and navigation during image-guided puncture.

• The system offers advantages during punctures performed by beginners and allows short procedure times for experienced interventionalists.

• The use of ultrasound as a live imaging modality reduces radiation exposure for medical staff.

• Providing information directly in the radiologist’s field of view improves ergonomics during the procedure.

 

MissingLinkText

 

MissingLinkText

Conflict of Interest

Holo4Med employees were only involved in the installation and calibration of the equipment, the training of the study supervisors in the use of the equipment before the start of the study, the technical support and the providing of technical details. The study itself was performed and analyzed exclusively by persons who work in the clinic and neither receive any payment from Holo4Med or have an employment relationship with Holo4Med.

• References

• 1 Moosburner S, Remde C, Tang P. et al. Real world usability analysis of two augmented reality headsets in visceral surgery. Artificial Organs 2019; 43: 694-698
• 2 Elmi-Terander A, Nachabe R, Skulason H. et al. Feasibility and Accuracy of Thoracolumbar Minimally Invasive Pedicle Screw Placement With Augmented Reality Navigation Technology. Spine (Phila Pa 1976) 2018; 43: 1018-1023
• 3 Scherl C, Stratemeier J, Rotter N. et al. Augmented Reality with HoloLens in Parotid Tumor Surgery: A Prospective Feasibility Study. ORL J Otorhinolaryngol Relat Spec 2021; 83: 439-448
• 4 Al Janabi HF, Aydin A, Palaneer S. et al. Effectiveness of the HoloLens mixed-reality headset in minimally invasive surgery: a simulation-based feasibility study. Surg Endosc 2020; 34: 1143-1149
• 5 Jiang T, Yu D, Wang Y. et al. HoloLens-Based Vascular Localization System: Precision Evaluation Study With a Three-Dimensional Printed Model. J Med Internet Res 2020; 22: e16852
• 6 Okachi S, Matsui T, Sakurai M. et al. Real-time ultrasound-guided thoracentesis simulation using an optical see-through head-mounted display: a proof-of-concept study. J Ultrason 2024; 24: 20240012
• 7 Wauben LSGL, van Veelen MA, Gossot D. et al. Application of ergonomic guidelines during minimally invasive surgery: a questionnaire survey of 284 surgeons. Surg Endosc 2006; 20: 1268-1274
• 8 El Shallaly G, Cuschieri A. Optimum view distance for laparoscopic surgery. Surg Endosc 2006; 20: 1879-1882
• 9 HoloLens 2—Overview, Features, and Specs | Microsoft HoloLens. o. J..
• 10 Park BJ, Hunt SJ, Nadolski GJ. et al. Augmented reality improves procedural efficiency and reduces radiation dose for CT-guided lesion targeting: a phantom study using HoloLens 2. Sci Rep 2020; 10: 18620
• 11 Solbiati M, Passera KM, Rotilio A. et al. Augmented reality for interventional oncology: proof-of-concept study of a novel high-end guidance system platform. Eur Radiol Exp 2018; 2: 18
• 12 Bettati P, Chalian M, Huang J. et al. Augmented Reality-Assisted Biopsy of Soft Tissue Lesions. Proc SPIE Int Soc Opt Eng 2020; 11315: 113150W
• 13 Damm R, El-Sanosy S, Omari J. et al. Ultrasound-assisted catheter placement in CT-guided HDR brachytherapy for the local ablation of abdominal malignancies: Initial experience. Fortschr Röntgenstr 2019; 191: 48-53
• 14 Farshad-Amacker NA, Kubik-Huch RA, Kolling C. et al. Learning how to perform ultrasound-guided interventions with and without augmented reality visualization: a randomized study. Eur Radiol 2023; 33: 2927-2934
• 15 Costa N, Ferreira L, de Araújo ARVF. et al. Augmented Reality-Assisted Ultrasound Breast Biopsy. Sensors (Basel) 2023; 23: 1838
• 16 Pfefferle M, Shahub S, Shahedi M. et al. Renal biopsy under augmented reality guidance. Proc SPIE Int Soc Opt Eng 2020; 11315: 113152W
• 17 Li H, Yan W, Zhao J. et al. Navigate biopsy with ultrasound under augmented reality device: Towards higher system performance. Computers in Biology and Medicine 2024; 174: 108453
• 18 Bettati P, Young J, Rathgeb A. et al. An augmented reality-guided biopsy system using a high-speed motion tracking and real-time registration platform. Proc SPIE Int Soc Opt Eng 2024; 12928: 129281G
• 19 Lin MA, Siu AF, Bae JH. et al. HoloNeedle: Augmented Reality Guidance System for Needle Placement Investigating the Advantages of Three-Dimensional Needle Shape Reconstruction. IEEE Robotics and Automation Letters 2018; 3: 4156-4162
• 20 Nicolau SA, Pennec X, Soler L. et al. An augmented reality system for liver thermal ablation: Design and evaluation on clinical cases. Medical Image Analysis 2009; 13: 494-506

Publication History

Received: 08 April 2024

Accepted after revision: 05 September 2024

Article published online:
04 October 2024