Keywords interventional procedures - biopsy - ultrasound - interventional radiology - augmented
reality
Introduction
Augmented reality (AR), which allows additional information to be overlaid on top
of reality, has many benefits beyond interventional radiology. For example, additional
information such as CT or MRI images can be made available directly in the field of
view during the intervention. This includes three-dimensional holograms of the patient’s
organs or 3D navigation data of a puncturing needle. Head-mounted displays (HMDs)
are widely used to display such AR images. One such HMD is the Microsoft HoloLens
2. The potential use of AR in medicine has already been demonstrated in other studies
for many specialties. These include general surgery [1 ], orthopedics [2 ], thyroid surgery [3 ], urology [4 ], and vascular surgery [5 ].
CT fluoroscopy, which is currently used for complex punctures, is a challenging procedure
that requires a high degree of spatial awareness on the part of the interventionalist.
The position of the needle in the three-dimensional body must be abstracted from two-dimensional
axial CT images and can only be supported to a limited extent by multiplanar reconstructions.
Augmented reality could be a suitable method to simplify this orientation. By displaying
3D projections and the associated depth perception, better transferability to the
patient's body can be achieved [5 ]. This simplifies puncture with alternative access routes, which could minimize the
risk of injury to critical structures and thus increase patient safety.
Studies in the field of urology have already shown that augmented reality can significantly
minimize procedure time and achieve better quality results [4 ]. Shorter procedure times could help to compensate for the increasing workload with
punctures.
AR has many additional benefits. For example, data (3D models, live images, etc.)
can be projected directly into the user’s field of view. For example, it is possible
to combine fusion images from CT and other modalities with 3D holograms of the patient’s
organs. In preparation for biopsies, it offers the possibility of planning puncture
paths on three-dimensional models and displaying them during the procedure. Combined
with live tracking of needles and other instruments, a new and cost-effective navigation
system for minimally invasive interventions can be realized. Importantly, the information
does not need to be displayed on an additional screen but can be projected directly
onto the HMD in the user’s field of view. This leads to improved ergonomics during
the procedure and a focus on the essential content [6 ]. By eliminating the need for additional screens, which often have to be positioned
in inconvenient locations, associated problems such as back, shoulder, and neck pain
can be avoided [7 ].
It also minimizes the otherwise increased risk of iatrogenic injury due to a disrupted
visual-motor axis [8 ].
An HMD can be worn under sterile conditions [3 ]. This offers the possibility of using the navigation system during a procedure,
potentially reducing the frequency and duration of CT fluoroscopy and thus radiation
exposure, especially for medical staff.
The primary objective of this study was to determine the acceptance and clinical feasibility
of a HoloLens 2-based AR system for minimally invasive CT-guided interventional radiology
procedures. The secondary objective was to evaluate the learning curves of subjects
with different levels of experience.
Material and methods
Hardware
An overview of the material used can be seen in [Fig. 1 ].
Fig. 1 Equipment.
A HoloLens 2 (version: 20348.1542, Microsoft Corporation, Redmond, USA) was used as
the AR system. This can be controlled entirely by hand gestures to allow operation
under sterile conditions. The HoloLens 2 displays have a 2K resolution in a 3:2 format,
which corresponds to a resolution of >2500 light points per radian [9 ]. The HMD was connected to the workstation via a 5 GHz Wi-Fi network.
A stereo camera (MicronTracker 3 Hx40, ClaroNav Inc., Toronto, Canada) was used for
optical tracking.
The punctures were performed on a phantom (CIRS triple modality 3D abdominal phantom,
Model 057A, Sun Nuclear Corporation, Melbourne, USA) with internal structures (ribs,
spine, kidneys, liver, hepatic vein, lungs), which allows both ultrasound and CT imaging
([Fig. 2 ]). Needles with a working length of 150 mm (17G, 1.4 × 180 mm, KLS Martin SE & Co.
KG, Tuttlingen, Germany) were used for the puncture simulation.
Fig. 2 CT scan of phantom.
For optical tracking, six optical trackers with a small metal ball in the optical
center were attached to the phantom. This ball can be detected on CT, so that the
CT data can later be fused with the optical images from the camera. Special optical
markers (Holo4Med S.A., Białystok, Poland) were also used to track the needle. A curved
array probe (3–11 MHz) (S40, SonoScape Medical Corp., Shenzhen, China) or a C5–2 probe
(2–5 MHz) (ACUSON Freestyle, Siemens Healthineers AG, Forchheim, Germany) was used
for sonography.
Software and application:
The software that was used was an application called HoloMIAI (Holo4Med S.A., Białystok,
Poland). The DICOM files of the phantom were converted by the software into a 3D model
with segmented internal structures. In this model, needle paths can be planned by
defining entry and target points. The HoloLens 2 displays the 3D model of the phantom,
including the planned needle trajectories, to the subject. A line is displayed between
the planned entry point and the target point, which extends out of the phantom for
easy orientation. During the puncture, the puncture needle is also projected onto
the 3D model and must overlap with the extended puncture line of the 3D model in order
to puncture the target structure correctly. In addition, the target point changes
color from red to green as soon as the extended needle tip is pointed at it ([Fig. 3 ] and [Fig. 4 ]). Furthermore, the subjects have another tool at their disposal called “Aim-Panel”.
This is an aiming guide consisting of a red ring, a blue ring, and a white dot. For
a correct puncture, both rings must be placed over the white dot. The blue ring indicates
the distance of the needle tip to the entry point and the red ring indicates the correct
alignment of the needle tip to the target. In addition, the distance to the target
structure is indicated by a bar that fills as the target is approached ([Fig. 3 ] and [Fig. 4 ]). At a distance of approximately 2 cm from the target, ultrasound should also be
used. This is projected into the user’s field of view in the HoloLens 2 as well and
serves as a live imaging modality during the final puncture ([Fig. 4 ]). All three components (3D model, Aim-Panel, and ultrasound image) can be freely
positioned in the room by the user using hand gestures and can be shown or hidden
as desired.
Fig. 3 AR image of the HoloLens with the needle correctly aimed at the target structure.
Fig. 4 AR image during the final puncture with ultrasound.
Study program
The acceptance and feasibility of the AR system was evaluated in subjects with different
levels of experience and age (3 medical students, 2 residents, 3 specialists, 1 senior
physician, and 1 chief physician). In addition, the subjects’ professional experience
was recorded in the form of years in radiology and they were asked to state the number
of CT fluoroscopies, ultrasound examinations, and ultrasound-assisted punctures they
had performed. Previous experience with AR or VR (virtual reality) was also documented.
All subjects first received an introduction to the AR system and underwent the general
tutorial of the HoloLens 2 from Microsoft, in which they learned about the general
operation. The HoloLens was also calibrated to the subject’s eyes. The subjects were
then given the opportunity to familiarize themselves with the software. They were
first given an introduction to the various tools and options of the HoloMIAI system
and how to use them. They were also given an introduction to the main voice commands.
These were also available as a list during the tests. After the subjects had familiarized
themselves with the operation of the system, a practice phase with a total of 3 punctures
followed: a first practice puncture under standardized instructions in a target region
6 cm deep and approximately 8 mm in diameter, followed by two further punctures with
different needle paths and target sizes for training purposes ([Fig. 5 ]).
Fig. 5 User during puncture.
After the training phase, the test subjects were asked to puncture five different
target lesions of different sizes and with different lengths and angulations of the
needle path, avoiding the critical structures of the model (lung and hepatic vein).
The diameters of the lesions varied between 7 mm and 15 mm and the length of the needle
path between 65 mm and 143 mm.
The needle tip positions were then documented and evaluated using a CT scan. The time
required from needle insertion for the first puncture until the subject declared the
completion of the fifth lesion puncture was documented. If not all target lesions
were hit correctly, a new round of 5 punctures was performed. Any problems and their
reasons were also documented.
After the puncture, the subjects were asked to complete a questionnaire on the clinical
feasibility, safety, and handling of the system. The subjects were asked to rate questions
([Table 1 ]) on a Likert scale from ‘strongly agree’ (5) to ‘strongly disagree’ (1).
Table 1 Questionnaire regarding clinical applicability, safety, and handling with results.
Questions
N
Mean
Standard deviation
Median
Minimum
Maximum
I can imagine using HoloLens in everyday clinical practice.
10
4.60
.516
5.00
4
5
The system simplified orientation during the intervention.
10
4.30
.949
5.00
3
5
The system simplified needle navigation.
10
4.70
.675
5.00
3
5
I quickly got used to using the system.
10
3.80
.919
4.00
2
5
Operation of the HoloLens was user-friendly.
10
3.90
.738
4.00
3
5
I found the display of information in the field of vision useful.
10
4.40
.516
4.00
4
5
The image quality of the HoloLens was sufficient for the intended purpose.
10
4.70
.483
5.00
4
5
Puncture outside the CT gantry simplified the procedure. (physicians only)
7
4.57
.787
5.00
3
5
While using the system I felt secure.
10
3.60
1.075
4.00
2
5
Augmented reality should play a role in interventional radiology in the future.
10
4.50
.707
5.00
3
5
Augmented reality should play a role in the training of young physicians.
10
4.90
.316
5.00
4
5
I feel confident using the system.
10
3.20
1.033
3.00
2
5
I would like to use the system for percutaneous procedures on patients.
10
4.40
.699
4.50
3
5
I still need more training with the system.
10
4.20
1.317
5.00
1
5
Statistical analysis
IBM SPSS Statistics 28.0 (IBM Corp., Armonk, NY, USA) was used for statistical analysis.
First, a descriptive survey of the recorded characteristics was conducted. For qualitative
characteristics, frequencies were calculated and presented as absolute numbers and
percentages. For quantitative characteristics, measures of location and dispersion
were determined. Spearman’s rank correlations were used to examine correlations between
quantitative variables. These were interpreted according to Cohen (1988): |rs| = 0.10
– weak correlation, |rs| = 0.30 – moderate correlation, |rs| = 0.50 – strong correlation).
All tests were two-sided, and a p-value ≤ 0.05 was considered statistically significant.
Results
9 out of 10 subjects were able to hit all 5 target structures on the first attempt.
Only one resident needed a second round. On average, the subjects needed 29:39 minutes
for 5 successful punctures. It was not possible to examine the distance to the lesion
center, as the measurement inaccuracy would be too high for target lesions measuring
only 3.5 mm in radius, even with 1 mm CT slices because of 1.4 mm thick needles and
significant metal artifacts. Therefore, only the classification as “hit” or “no hit”,
which is relevant in practice, was used. There was a significant correlation between
the number of years in radiology and the time required for the punctures (rs =–0.787; p=0.007), as well as between the time required and previous experience with
CT fluoroscopy (rs =–0.755; p=0.012), sonography (rs =–0.632; p=0.050) and sonography-assisted punctures (rs =–0.745; p=0.013). Three of the subjects reported that they had once experienced VR
or AR outside of a medical context. However, this had no detectable connection with
a faster puncture time.
Medical students took an average of 43:00 min, residents 34:30 min, specialists 22:10
min, senior physicians 17:00 min, and chief physicians 15:00 min ([Fig. 6 ] and [Table 2 ]).
Fig. 6 Graphical representation of the time required for 5 punctures over the experience
level.
Table 2 Times required for 5 successful punctures.
Experience level
N
Mean
Standard deviation
Standard error of the mean
Median
Minimum
Maximum
Chief physician
1
15:00
.
.
Senior physician
1
17:00
.
.
Specialist
3
22:10
05:45
03:19
22:00
16:30
28:00
Resident
2
34:30
20:30
14:29
34:30
20:00
49:00
Medical student
3
43:00
02:00
01:09
43:00
41:00
45:00
Total
10
29:39
13:24
04:14
25:00
15:00
49:00
In the survey of participants, mainly positive and largely supportive statements were
collected. As part of the questionnaire with a Likert scale of 1 to 5 points, the
mean value (M) of the answers given was calculated. The test subjects stated that
they could imagine using the HoloLens in everyday clinical practice (M=4.6), that
the system simplified needle navigation (M=4.7), that the image quality was sufficient
for the intended purpose (M=4.7), and that puncture outside the CT gantry simplified
the procedure (M=4.57). The user-friendliness and quick familiarization with the handling
were rated with an average of M=3.9 and M=3.8, respectively. The question of whether
the subject felt safe during the puncture was rated with M=3.6. With an average rating
of M=4.2, the test subjects agreed that they would like to use the system for percutaneous
procedures on patients in the future. The need for further training was also rated
at M=4.2 ([Table 1 ]). In general, there was no recognizable correlation between the scores given by
medical students and more experienced test subjects. The only striking finding was
that the chief physician surveyed already felt so confident with the system that he
rated the need for further training as 1 and thus differed considerably from the rating
of the other test subjects (M=4.2).
Overall, technical problems were rare. For two subjects, the HoloLens switched off
due to overheating. This was presumably due to the additional computational load caused
by the live-view of the HoloLens image to a PC during the study. There were two short
transmission problems with the Wi-Fi, and one subject’s needle marker bent slightly,
resulting in incorrect tracking of the needle.
Discussion
The results show that the AR system has many advantages for percutaneous procedures
in interventional radiology, for example, improved orientation during the intervention,
which is particularly advantageous for needle navigation. In addition, the integration
of live ultrasound imaging while maintaining the possibility of CT fluoroscopy has
the potential to drastically reduce radiation exposure for staff without reducing
the safety of needle navigation in patients.
A significant correlation between professional experience and previous experience
in interventional radiology of the test subjects and the time required for the punctures
was to be expected and was confirmed here. Nevertheless, it should be emphasized that
all medical students, without any professional experience, were also able to successfully
perform the punctures with this system in the first attempt. This advantage in percutaneous
procedures for completely inexperienced medical students is in line with results with
other AR systems (without US) [10 ]. This shows the potential to flatten the learning curves for percutaneous procedures
in interventional radiology and thus enable the training of more interventionalists
through faster training.
User acceptance of the system was very high and handling was perceived as intuitive.
In addition, all of the test subjects surveyed could well imagine using the system
in everyday clinical practice. However, 80% of the test subjects stated that they
needed further training with the system, which probably also explains the relatively
lower rating regarding feeling safe during the puncture.
The applicability of 3D models displayed using augmented reality for punctures has
been confirmed in other studies [10 ]
[11 ]
[12 ]. Studies have also already shown the applicability of ultrasound during CT-guided
percutaneous procedures, with a significant reduction in radiation exposure [13 ].
However, this study is one of the first to evaluate a system that combines optically
assisted needle tracking with AR projection of a 3D model and live ultrasound displayed
in the user’s field of vision for percutaneous procedures. The superiority of a US
image displayed using AR for percutaneous biopsies, particularly due to improved ergonomics
[6 ], more comfortable working [14 ], and improved precision [15 ], has already been confirmed in other studies. The possibility of combining US and
CT images for AR-assisted punctures [16 ], as well as instrument tracking and AR-projected ultrasound, has also already been
demonstrated [17 ]. The applicability of a system comparable to the one tested here, with 3D navigation
and ultrasound, has already been demonstrated for prostate punctures [18 ].
The average puncture time of 5:56 min/puncture with the AR/US procedure tested here
in a phantom with the hepatic vein and lung as the main risk structures is comparable
with the results of other studies with AR-based puncture systems. Here, the puncture
times in a phantom without any risk structures with AR navigation in combination with
CT fluoroscopy were 4:42 min/puncture [10 ] and 9:24 min/puncture for purely AR-supported punctures in a human cadaver [11 ]. It should be noted that the systems compared here differ significantly in terms
of technology, but there are indications that the puncture time increases with the
increase and complexity of risk structures.
One challenge is the usual deformation or bending of the needle in the phantom or,
in the future, in the patient and the associated, not 100% straight puncture path,
as the needle tip is abstracted from the marker at the end of the needle using a straight
line as part of optical tracking. Therefore, continuous monitoring by live imaging
modalities such as ultrasound is required, especially for the last 2–3 cm of the puncture
path. It would also be conceivable to connect the system to bend-sensitive needles
[19 ].
Another important limitation is that this is a stationary phantom that has no respiratory
movement. This problem could be minimized by combining it with respiratory movement
monitoring [20 ]. It is also conceivable that the artifacts of optical tracking caused by respiratory
movement could be compensated by performing the final puncture under live imaging.
In patients or target structures where imaging by ultrasound is not possible, live
imaging of the final end segment by CT fluoroscopy could also be considered [10 ].
In combination with ultrasound, the use of the system is limited to puncture sites
where ultrasound is possible. It is, therefore, particularly suitable for punctures
in the abdomen, for example for liver or kidney punctures. CT fluoroscopy should always
be available if an adequate ultrasonic window cannot be found.
Based on these promising properties, we have already initiated a randomized clinical
trial (prospective evaluation of an AR-based procedure for percutaneous procedures
in interventional radiology).
In summary, the system has high potential for practical application, in particular
the potential reduction of radiation exposure while maintaining safety through live
imaging, as well as the possible improvement of ergonomics and orientation could possibly
lead to a more efficient, user-friendly, and safer intervention.
Clinical relevance
The AR system provides improved orientation and navigation during image-guided puncture.
The system offers advantages during punctures performed by beginners and allows short
procedure times for experienced interventionalists.
The use of ultrasound as a live imaging modality reduces radiation exposure for medical
staff.
Providing information directly in the radiologist’s field of view improves ergonomics
during the procedure.
