Key words
artifical intelligence - oncological imaging - hybrid imaging
Introduction
Artificial intelligence (AI) applications are believed to provide promising tools
for the analysis of evolving multi-omics data in diagnostic medicine [1]. With significant methodological advances in AI, applications continue to improve
and may support experts in task-specific applications [2]. However, to date, AI applications cannot replace physicians in complex tasks that
require human-guided decisions and interactions. While moving through the hype cycle
with respect to expectations towards AI, the initial euphoria is currently being dampened
by studies with a clear focus on limitations and weaknesses [3]. Due to the level of digitalization and natural accruement of big data, AI applications
might add value to patient care by assisting physicians in simple tasks [4]
[5]. With disproportionate growth of imaging data in a single examination, measures
to increase productivity and to leverage data are desired to assist physicians and
technicians with pre-screened data, optimizable raw images/post-processing tools,
and quantitative features along the whole imaging workflow [6]. Hybrid imaging is the combination of morphological (CT/MRI) and functional imaging
using a variety of radiotracers such as 18F-FDG, 68Ga/18F-PSMA, or 18F-SIFA-TATE, and provides complementary information regarding, e. g., tumor characteristics
and metabolism. This method generates huge datasets with only portions of data such
as standardized uptake values (SUV) or tumor size currently being used. Deep learning
might benefit from the application of artificial intelligence in automated raw image
pre-processing to refine image quality with promising applications in ultra-low radiation
imaging, attenuation correction, and de-noising [7]
[8]. Further areas of implementation include image reconstruction, image processing,
and automated image analysis by machine learning approaches [9]. The main clinical applications of radiomics might apply in a diagnostic context
to image-based applications including particularly time-consuming tasks such as lesion
detection, characterization, and monitoring [9] ([Fig. 1]), which can be considered challenging with increasing numbers of tumor manifestations,
examination time points, and heterogeneity of tumor burden [9]. The impact of whole tumor burden heterogeneity assessment in imaging studies will
likely evolve as an interesting field of study since biopsies are prone to sampling
error that is hardly addressed in current practice [10]. Beyond diagnostics, accurate delineation and segmentation tasks play a crucial
role in radiotherapy treatment planning [11].
Fig. 1 Applications of AI in oncological imaging along the radiology workflow: a Detection of lesions in schematic drawings of patients with lung cancer, prostate
cancer, and neuroendocrine tumor, b Characterization of solitary lesions in axial PET/CT reconstructions of lung cancer
and prostate cancer, additional circles drawn to highlight the areas of interest.
c Longitudinal monitoring of single lesions with regard to aforementioned characteristics
allowing response assessment (axial PET-reconstruction) in lung cancer with changing
characteristics over time.
Hybrid imaging including PET/CT and PET/MRI provides complementary imaging data allowing
a multifaceted anatomical, functional and molecular characterization of tumor manifestations.
With different radiotracers, hybrid imaging is applied in most malignant tumors including
lung cancer, prostate cancer, and neuroendocrine tumors with significant effects on
patient management compared to conventional imaging algorithms as reflected by German
and international guidelines [12]
[13]
[14]. The limitations of single-modality imaging are overcome by the strengths of the
complementary modality, e. g. superior lesion detection in PET and superior anatomical
resolution in CT or MRI, with consecutively significantly increased diagnostic accuracy
[15].
While the benefits of automated tumor delineation, tumor characterization, or tools
for longitudinal tumor volume monitoring are intuitive, AI-enhanced analysis methods
facilitate the extraction of more subtle information from imaging data that are mostly
undetectable for human readers with deep learning approaches. As of now, the majority
of AI studies in hybrid imaging are retrospective, with a lack of clinical translation
due to unresolved limitations in algorithm development, validation, and clinical implementation
[16]. Additionally, prospective trials on the applications of artificial intelligence
are scarce and therefore do not allow for general translation of results on larger
cohorts and further indications. Multicenter, randomized controlled, prospective trials
will be necessary to demonstrate the usefulness of radiomics and to scrutinize retrospective
results of radiomics and AI imaging studies [17].
Prospectively, oncological applications of AI beyond imaging-based tasks will focus
on a holistic integration of multi-source diagnostic data including radiomics, genomics,
and metabolomics to personalize diagnostics at the molecular, cellular, and organism
level [4]
[18]
[19]. In this narrative review, we provide an overview of AI applications in oncological
hybrid imaging. The first part provides an introduction into the principles of AI
for the analysis of medical imaging data with review of the most recent literature
related to applications in oncological hybrid imaging. The second part discusses AI
applications in hybrid imaging in lung cancer, prostate cancer, and neuroendocrine
tumors.
Technical realization, basics of data acquisition, and analysis
Technical realization, basics of data acquisition, and analysis
Radiomics
The combination of automated quantitative image analysis with supervised machine learning
(ML) is often referred to as radiomics ([Fig. 2]) [20]
[21]. Quantitative analysis describes image-based features with regard to tumor shape,
distribution of intralesional signal intensities (often referred to as histogram statistics),
and texture, i. e., spatial relationships of voxels and their respective grayscale
patterns, resulting in a large, high-dimensional feature space. Since many of these
image features may show strong correlations, a feature selection step may be helpful
to reduce the number of intercorrelated image features.
Fig. 2 Applications of AI in radiomic classification and deep learning algorithms. The first
row shows the radiomics approach, comprising tumor segmentation, extraction of handcrafted
features, and training of an ML model. The bottom row illustrates an automated approach
of a deep learning algorithm with convolutional neural networks.
This image-derived feature space can then be linked to clinical outcomes, such as
diagnosis, prognosis, or treatment response, by fitting or training statistical or
machine learning models to the data. Ultimately, trained models may then be used to
predict clinical outcomes from imaging features. Popular models for these applications
are generalized linear models, support vector machines, and random forests [9].
Deep Learning Algorithms
Deep learning (DL) methods have gained considerable interest within medical imaging
research. In DL the algorithm learns a composition of features that reflect a hierarchy
of structures in the data [22]. These systems allow leveraging of the compositional nature of images by an end-to-end
approach integrating image-based features [22]. In contrast to traditional ML approaches, deep learning (DL) models based on convolutional
neural networks (CNN) do not require a predefined definition of image features but
are able to learn relevant features directly from imaging data [6]
[9]
[23]. Beyond the prediction of patient outcome, DL is particularly useful for object
detection, e. g. localization of lung nodules, or image segmentation for the assessment
of tumors or organs. The quality and quantity of imaging data for training and validation
play a pivotal role in the clinical application of DL models. Due to the high number
of free parameters that need to be determined in model training, DL models are particularly
data-hungry and require large amounts of curated and possibly expert-annotated imaging
data. Since DL models can easily be overfitted to training data, the reliability of
trained models and the quality of predictions need to be assessed and validated carefully
on independent data sets which are not used during training. A high level of evidence
is reached by validation on entirely independent data [16].
Hybrid imaging and machine learning
From a technical perspective, the fusion and integration of imaging modalities such
as PET and CT/MRI from hybrid imaging is straightforward, when images from both modalities
are sufficiently aligned – either through simultaneous acquisition or image registration.
In radiomics approaches, quantitative image features can then be calculated from each
image contrast separately and extend the feature space, i. e., the number of features
derived from each tumor or metastasis. Likewise, in a CNN approach, image contrasts
can be combined in channels – much like in the case of red, green, and blue channels
for standard photographic images. In both approaches, the increased information content
of the enlarged feature space needs to be accounted for in the statistical modelling
approach. An enlarged feature space requires models that can store more information
resulting in greater susceptibility to overfitting or memorizing of the training data.
Therefore, rigorous model validation plays a crucial role in hybrid imaging.
To fully exploit the potential of complementary hybrid imaging information, several
tumor co-segmentation methods were proposed in hybrid imaging [15]. Potenzial clinical applications will include identification of dedifferentiation
patterns as observed in neuroendocrine tumors or malignant transformation of lymphoma
that are associated with a worse prognosis [24]
[25]. Another highly relevant application will be lesion characterization, for instance
of lymph nodes with central necrosis, that can be associated with a worse prognosis
in a variety of malignancies including sarcoma [26].
Applicability of AI for imaging data optimization
AI applications were recently successfully evaluated for attenuation correction, pre-
and postprocessing, co-registration of data, and PET or MRI/CT-based motion correction.
A detailed review of potential applications can be found in [6].
Limitations, challenges, and perspectives
To meet the high standards in medical imaging, a variety of challenges will have to
be addressed to realize and accelerate the clinical translation of relevant AI applications
in medical imaging. Unambiguous nomenclature and clear definition of intended use
of models are prerequisites for broad implementation to differentiate mere data mining
approaches from (semi-) automated task-based applications and to fully integrate tools
into the medical imaging value chain [17].
From a radiomics perspective, processing of both small cohorts and unstructured big
data will not yield sufficiently robust algorithms to overcome unresolved obstacles
in standardization such as a lack of protocol harmonization and data heterogeneity
[27]. While small data sets will eventually lead to overfitting of algorithms, unstructured
big data sets will be insufficient for training purposes, thus generating inaccurate
algorithms. Therefore, the population for training and validation purposes must be
sufficiently powered, well-balanced, and organized with regard to complexity and application-relevant
features.
With significant preanalytical heterogeneities, harmonization of PET imaging remains
challenging. Divergences may for instance originate in a broad spectrum of applied
dose, distinctive physiological uptake patterns, different attenuation correction
methods, and scanner-related differences in image acquisition and reconstruction algorithms
[16]. From our experience, this compares to morphological imaging protocols including
both CT and MRI, with a relevant spectrum of applied contrast agents, time between
application of contrast agents, and modality parameters. Technical factors and reconstruction
algorithms have a substantial impact on the quality of the extracted radiomics features
and need to be considered [28]. Systematic methodical flaws need to be identified using independent external validation
providing meaningful performance metrics [17]
[29]. For this purpose checklists for the development and evaluation of artificial intelligence
tools in medical imaging have been designed to improve the quality of studies [30]
[31].
At this point, integrated AI applications for the structured analysis of hybrid imaging
data remain scarce, particularly due to data privacy and the lack of publicly available,
expert-annotated hybrid imaging data sets [6]. Structured data repositories, such as The Cancer Imaging Archive (TCIA), and large-scale,
privacy-preserving initiatives, such as the Radiology Cooperative Network (RACOON),
promise to increase sample sizes for the development of AI models, possibly with federated
learning approaches [27].
From an ethical and medicolegal perspective, AI applications in medical imaging will
require detailed explanation regarding development and codebase with proof of validation
and safety studies for approval before broad clinical implementation. In Europe, medical
devices including AI-based algorithms are not approved by a centralized agency and
will be regulated depending on their risk potential. While high-risk devices (IIa,
IIb, and III) are handled and certified by accredited, notified bodies in Europe,
low-risk devices will be released at the sole responsibility of the manufacturer and
ultimately also the user [32]. In contrast, in the US medical devices including AI-based algorithms are cleared
in three pathways: the premarket approval pathway (for risk-associated devices), the
de-novo premarket review (for low and moderate-risk devices), and the 510(k) pathway
[32]. With regard to the spectrum of approved AI applications, Luchini et al. reported
that a total of 71 oncology-related AI applications had been approved by the FDA as
of the May 31, 2021, with 39 (55 %) applications in cancer radiology [33]. Of these the majority are intended as an integrative application, potentially representing
the decisive step in the diagnostic workflow of cancer patients, with only one application
for de-noising of PET-images in hybrid imaging [33]. Finally, from a clinical workflow perspective, AI products will require seamless
integration in the diagnostic workflow with transparent and explainable results to
support decision making.
Applications of AI in oncological hybrid imaging
Applications of AI in oncological hybrid imaging
The following chapter reviews relevant clinical applications of AI in more detail
for lung cancer, prostate cancer, and neuroendocrine tumors, where hybrid imaging
has a significant impact on therapy guidance and clinical decision making in a tertiary
medical center. Further successful fields of AI application of PET/CT and PET/MRI
include lymphoma [34]
[35], breast [36] and brain cancer [37], cervical cancer [38] for which a multitude of studies report relevant clinical findings, e. g., correlations
of SUVmax of the primary breast tumor and significantly more frequently local recurrence in
surveillance [36].
Lung cancer
According to the German S3-guideline, hybrid imaging with 18F-fluorodeoxyglucose (18F-FDG) PET/CT is an established standard in the diagnostic algorithm of both small-cell
lung cancer (SCLC) and non-small cell lung cancer (NSCLC) [39]. Compared to conventional CT, 18F-FDG PET/CT provides significantly improved delineation of the primary tumor and
accurate assessment of metastases compared to conventional CT, detecting unexpected
lesions with significant effects on therapy management in 20–25 % of cases [40]
[41]. Currently, well-established single-modality AI applications exist in CT for pulmonary
nodule detection, with impact of lesion size and quantity on T- and eventually M-stage
classification and therapy planning in lung cancer [42]. Pulmonary nodules have a variety of CT-attenuation patterns with well-defined margins
between solid components and healthy lung parenchyma, unclear margins in subsolid
and ground-glass components, and even more heterogeneous borders in cases of associated
local infiltration, atelectasis, and pneumonic consolidation. Using an automated image
analysis approach, intrapulmonary lesions can be assessed with regard to different
characteristics [43]
[44]. The Dutch-Belgian lung cancer screening trial (NELSON), for example, was the first
screening trial to apply semi-automated computer aided-volumetry (CAV) instead of
handcrafted measurements, thereby achieving high negative predictive values and presumably
fewer false-positive results compared to other lung cancer screening trials [45]
[46]. Also PET-based single modality approaches have been studied and shown promising
results in segmentation of both pulmonary nodules and thoracic lymph nodes to predict
outcome [47].
However, there are also few well-documented examples of true multi-modality AI applications
in lung cancer. Wallis et al., for example, developed a deep learning method to detect
pathological mediastinal lymph nodes from whole-body 18F-FDG PET/CT. Model performance was comparable to that of an expert reader on data
from the same type of scanner, and transfer learning allowed translation to other
scanners [48]. Zhao et al. proposed a fully convolutional neural network on a cohort of 84 patients
with lung cancer who underwent 18F-FDG PET/CT showing that co-segmentation networks can combine the advantages of two
modalities effectively outperforming single-modality applications [15]. From a clinical perspective, this approach appears valuable in assessing the primary
pulmonary malignancy to guide T-stage classification. Beyond segmentation, multi-modality
applications have been shown to impact lesion characterization and prognostication.
In a retrospective multi-institutional study, Mu et al. showed that a radio-genomic
deep learning approach can be used to predict EGFR status with weak but significant
inverse correlation to PD-L1 status for noninvasive decision support in NSCLC [49]. The algorithm yielded an area under the receiver operating characteristics curve
of 0.81 with an accuracy of 78.5 % in an external test cohort of 65 patients with
higher performance when integrating anatomical and metabolic information compared
to single-modality approaches [49]. Yet, due to the limited ROC, physicians may not fully omit biopsy as a tool to
guide treatment on a patient level, notably when deciding for or against a certain
treatment. Further studies are required to improve the performance of these algorithms
to safely guide treatment selection.
Prostate cancer
Hybrid imaging with prostate specific membrane antigen (PSMA) ligands has gained broad
application in prostate cancer including biochemical recurrence, primary staging in
high-risk disease (Gleason score > 7, PSA > 20 ng/mL, clinical stage T2c-3a), and
response assessment with significant impact on clinical decision making, particularly
in the detection of metastatic lymph nodes and bone metastases [50]
[51]
[52]
[53]. In this context, most AI applications focus on single-modality approaches for lesion
detection. Kostyszyn et al. developed a CNN approach based on 68Ga-PSMA PET to assess intraprostatic gross tumor volume in a multi-center study of
152 patients with retrospective histopathologic correlation [54]. Results demonstrated fast and robust auto-segmentation of the intraprostatic tumor
volume not only in 68Ga- but also in 18F-PSMA PET/CT compared to manual segmentation and semi-automatic thresholding, which
encouragingly shows translatability between differently labelled PSMA ligands [54]. In another study, an ML algorithm was trained on 72 prostate cancer patients for
lesion detection, analyzing 77 radiomic features in 68Ga-PSMA PET/ low dose CT to differentiate physiological from pathological radiotracer
uptake, resulting in high sensitivity (97 %) with lower specificity (82 %) due to
frequent misinterpretation of physiologic PSMA uptake in glands [55]. To assess whole-body tumor burden, a semi-automatic software package (qPSMA) for
68Ga-PSMA PET/CT was introduced and validated with high correlation between total lesion
metabolic volume and PSA levels [56]. Using this tool, patients with very high tumor load showed a significantly lower
uptake of 68Ga-PSMA-11 in normal organs confirming a tumor sink effect [57]. This has clinical implications, as similar effects might occur with PSMA-targeted
radioligand therapy, making this tool interesting for pre-therapeutic stratification.
Without exceeding the radiation dose limits for organs at risk, these patients might
potentially benefit from increased therapeutic activity [57]. In another single-center cohort of 83 patients, Moazemi et al. investigated deep
learning applications in pre-therapeutic 68Ga-PSMA PET/CT for lesion detection and 177Lu-PSMA therapy guidance in metastatic prostate cancer, showing high diagnostic accuracy
[58]. Radiomic features (SUVmin, SUV correlation, CT min, CT busyness and CT coarseness) in 68Ga-PSMA PET/CT and clinical parameters such as Alp1 and Gleason score yielded strong
correlations with changes in prostate-specific antigen (PSA) to predict outcome [58]
[59]. This finding also points in the direction of integrated diagnostics where the integration
of multi-source diagnostic data from medical imaging, pathology, liquid biopsy, and
clinical findings is analyzed to achieve optimized diagnostic accuracy in evidence-based
clinical decision guidance.
Only a few true hybrid deep learning applications have been evaluated in prostate
cancer imaging, including prediction and response assessment. Papp et al. developed
an ML approach to predict low vs. high lesion risk, biochemical recurrence, and overall
patient risk using 68Ga-PSMA PET/MRI with excellent cross-validation performance based on a cohort of 52
patients selected from a prospective randomized trial in primary prostate cancer [60]. The algorithm yielded 89 % and 91 % accuracy in biochemical recurrence and overall
patient risk, respectively. In this study, feature ranking demonstrated that molecular
68Ga-PSMA PET was the dominant in vivo feature source for lesion risk prediction compared
to MRI which yielded ADC but not T2w parameters as high-ranking features. The authors
hypothesized that integration of PSMA and ADC features in a model scheme could deliver
a superior predictive value [60]. However, notably the latter study may be difficult to interpret due to methodical
limitations such as lack of inter- and intrareader variability analysis and omission
of a final model with validation on an independent test set.
Neuroendocrine tumors
Neuroendocrine neoplasms (NEN) are a heterogeneous group of malignancies with a variety
of histological subtypes, primary location, and functional status. NEN are classified
as differentiated neuroendocrine tumors (NET) with preserved somatostatin receptor
(SSTR) status and poorly differentiated neuroendocrine carcinomas (NEC). Since NETs
usually progress slowly and several treatment options are available, the prevalence
of NETs is increasing along with the number of imaging examinations, which often show
significant metastatic tumor burden, impacting the radiologic workload [61]. To address these difficulties and to improve standard of care, the European Neuroendocrine
Tumor Society (ENETS) promotes structured reporting in radiology and molecular imaging
studies [62]
[63].
Hybrid PET/CT imaging with somatostatin-receptor agonists, such as 68Ga-DOTA-TATE, 68Ga-DOTA-TOC and most recently 18F-SiFAlin-TATE, allows longitudinal multimodal assessment of morphology and SSTR expression
in therapy guidance in NETs [64]. While tracer biodistribution of the established SSTR agonists is similar, minor,
yet existing differences in physiological distribution profiles may pose a challenge
for an automated image-segmentation approach. Single-modality AI applications have
shown their potential in the diagnostic workup to help distinguish pathology, aid
lesion detection, and facilitate response assessment in NETs. Criteria-based reporting
systems including RECIST 1.1 and the Krenning Score allow patient stratification in
a single-modality approach, with SSTR-RADS serving as an example for multimodal assessment
criteria, which could help structure the outcome of classifications-based algorithms
[65].
Promising results have been reported with respect to AI applications for grading in
both CT and MRI in preoperative morphological imaging studies [66]
[67]. Liberini et al. and Atkinson et al. provide convincing data to suggest that statistical
and histogram-based parameters of SSTR-ligand PET may have added value for prediction
and therapy response [68]
[69]. Recently, SSTR expressing tumor volume and total lesion SSTR expression were proposed
as first-order molecular prediction biomarkers assessed by AI [69]
[70]
[71]. Skewness and kurtosis of tumor lesions on pretreatment 68Ga-DOTA-TATE PET/CT were shown to predict responsiveness to radionuclide peptide treatment
[71]. However, these first-order features do not necessarily reflect true radiomic features
to use the hidden potential of imaging data. Wehrend et al. developed a DL algorithm
to automatically detect tumor lesions in 68Ga-DOTA-TATE PET in a study of 125 patients [72]. Despite promising results, high physiological liver uptake and comparably low spatial
resolution hamper the diagnostic accuracy of PET with SSTR analogs in the detection
of liver lesions, making hybrid imaging with MRI desirable. Fehrenbach et al. developed
a DL algorithm in gadoxetic-acid (Gd-EOB)-enhanced MRI for the assessment of hepatic
tumor burden in NEN based on an initial training cohort of 222 imaging studies. Their
application shows high accuracy in the detection and quantification of liver metastasis,
facilitating clinical decision making in multidisciplinary cancer conferences [61]. Taking both into account, it is likely that integration of complementary data streams
could refine AI algorithms. Yet most studies focus on automated assessment of hepatic
tumor burden in NEN with very limited available literature evaluating the performance
and added value of fused hybrid imaging features for therapy guidance in NETs.
Discussion and conclusions
Discussion and conclusions
The application of AI in hybrid medical imaging offers potential for the automated
delineation, noninvasive characterization, and longitudinal monitoring of oncological
diseases. Yet, many hurdles remain to be addressed before AI can be implemented in
daily routines. Validation of AI tools will require methodological approaches and
significant evidence with prospective trials to demonstrate the impact of AI tools
on patient outcomes [16]
[17]. In the near feature AI applications will more likely represent additional tools
rather than standalone diagnostic algorithms [28].
Undoubtedly, hybrid imaging has significant advantages regarding diagnostic accuracy
compared to its complementary standalone modalities but new challenges with respect
to data volume and structured analysis need to be overcome to fully exploit its potential
in the context of precision medicine [20]. Task-based applications in lung cancer, prostate cancer, and neuroendocrine tumors
indicate that technical implementation is feasible with significant impact on the
medical imaging work stream and may in the future provide down-stream clinical decision
support in precision oncology [73]
[74]. With innovative molecular and cellular oncological treatments, multi-modality applications
may impact therapy guidance by assessing complex therapy response patterns and metastatic
heterogeneity. AI applications present as transformative technology to supersede single-modality
algorithms for automated detection, noninvasive characterization, and longitudinal
monitoring of oncological disease in hybrid imaging [12]. However, true complementary multi-modality algorithms remain scarce with a majority
of applications being based on single-modality approaches. In addition to the aforementioned,
applications of artificial intelligence have also been evaluated in a variety of other
malignancies. In renal cell cancer, PET/MRI radiomic signatures analyzed in three
separate feature sets showed that the combined functional and structural information
of PET/MRI had a higher correlation with tumor microvascular density [75]. Additionally, promising results from using fusion models that integrate data from
CT/MRI/PET have been reported and showed better results than separate image analysis
[76].
To close the translational gap of AI applications in medical hybrid imaging, challenges
need to be addressed to improve safety, quality, and ultimately public trust [9]
[27]. While expectations regarding AI tools in medical imaging have become more critical,
by focusing on the limitations and weaknesses of the technology, we expect future
research and development to yield valuable task-based tools for medical imaging in
radiology and nuclear medicine.
