Introduction
Endoscopic ultrasonography (EUS) provides high resolution images of the pancreas,
and it is considered one of the most accurate methods for the diagnosis and staging
of chronic inflammatory, cystic, and neoplastic pancreatic diseases [1]
[2]. The differential diagnosis of solid pancreatic masses, however, remains a challenge
[3]. EUS can guide fine-needle aspiration (EUS-FNA) for obtaining cytologic samples
of pancreatic lesions, thus making a pathologic diagnosis possible [4]
[5]. EUS-FNA, however, may be technically demanding, and multiple punctures of pancreatic
lesions may be needed to obtain adequate material for cytologic or microhistologic
evaluation. Furthermore, EUS-FNA of the pancreas is associated with a small, but not
insignificant, morbidity [6]
[7]. In addition, the sensitivity of cytology for malignancy is limited, and false-negative
results are obtained in up to 20 % to 40 % of cases [8]
[9]. In an attempt to overcome these limitations of EUS-FNA, techniques of image enhancement
are currently under active technical development. Contrast-enhanced harmonic endoscopic
ultrasonography (CH-EUS) is one of the most promising in this context [10].
Contrast-enhanced harmonic ultrasonography is a method used for the real-time evaluation
of tissue perfusion, without Doppler-related artifacts, and has improved the depiction
and characterization of digestive diseases by trans-abdominal ultrasonography [11]
[12]. It is based on the detection of intravenous contrast agents by means of a dedicated
harmonic. Recently, it was found that contrast-enhanced harmonic imaging could be
generated by using an echo-endoscope with a wide-band transducer [13]. Recent studies have shown that CH-EUS may improve the evaluation of solid pancreatic
lesions (SPLs) and that the microvascular pattern closely correlates with the histologic
features of the lesion [14]
[15]. In a recent meta-analysis [16], the pooled sensitivity of CH-EUS for the differential diagnosis of pancreatic adenocarcinoma
was 94 % (95 % confidence interval [95 %CI] 0.91 – 0.95), and the specificity was
89 % (95 %CI 0.85 – 0.92). The area under the curve under summary receiver operating
characteristic (AUROC) was 0.9732. The pooled positive likelihood ratio was 8.09 (95 %CI
4.47 – 14.64), and the negative likelihood ratio was 0.08 (95 %CI 0.06 – 0.10).
Although CH-EUS seems to be promising for the evaluation of SPLs, it is not clear
whether the interpretation of CH-EUS is reproducible among different endosonographers.
Previous reports assessing the reproductibility of CH-EUS for the evaluation of SPLs
involved mainly experienced endosonographers. The main aim of this study was to assess
the interobserver agreement (IOA) of CH-EUS in the evaluation of SPLs by endoscopists
with different levels of experience in EUS and CH-EUS. We additionally evaluated the
accuracy of CH-EUS for the diagnosis of pancreatic adenocarcinoma by endoscopists
with different levels of experience in EUS and CH-EUS.
Participants and methods
Design of the study and selection of patients
This was a cross-sectional observational study with two aims. The primary aim was
to assess the IOA of CH-EUS in the evaluation of SPLs by endoscopists with different
levels of experience in EUS and CH-EUS. The secondary aim was to assess the accuracy
of CH-EUS for the diagnosis of pancreatic adenocarcinoma by endoscopists with different
levels of experience in EUS and CH-EUS.
A total of 60 patients with SPLs who underwent routine EUS at the Department of Gastroenterology,
University Hospital of Santiago de Compostela (Spain), during 2011 were consecutively
included in this study after giving informed consent for EUS. A final diagnosis of
malignant or benign tumor was defined according to the following reference methods:
(1) histologic findings of surgical specimens in patients undergoing surgery; (2)
cytologic findings definitely positive for malignancy, together with compatible EUS
and computed tomographic (CT) findings for a final diagnosis of malignant disease,
in patients with unresectable tumors; and (3) EUS and CT findings at entry, clinical
presentation, and a minimum follow-up period of 6 months, including EUS-FNA and CT,
for a final diagnosis of benign disease in patients with benign cytologic findings.
All of the material provided for the study was anonymous, and in no instance was a
patient’s identity revealed. The study was approved by the local institutional review
board and conducted in accordance with the Declaration of Helsinki and its amendments,
as well as Good Clinical Practice guidelines.
Technique of endoscopic ultrasonography and selection of videos
In each of the 60 patients, the SPL was evaluated with standard EUS imaging and CH-EUS. EUS
was performed with a linear EUS probe (EG3830UTK; Pentax Europe GmbH, Hamburg, Germany)
attached to a Preirus platform (Hitachi Medical Systems GmbH, Wiesbaden, Germany),
which includes the harmonic module. All procedures were done by two experienced endosonographers
(J. I.-G. and J. L.-N.). SonoVue (sulfur hexafluoride MBs; Bracco International BV,
Amsterdam, the Netherlands) was the contrast agent used for CH-EUS in all cases. The
complete description of the technique of CH-EUS has been reported elsewhere [10]. For each patient, one video sequence was recorded for 2 minutes, starting at the
time of SonoVue administration ([Video 1]). Each video sequence also included a B-mode standard EUS image of the lesion of
interest. Each video sequence was labeled with a random number by an endosonographer
who had not participated in the EUS procedure and was blinded to the clinical history
and the pathologic diagnosis. The observers were provided with a pen drive containing
the 60 video sequences and were allowed unlimited time to review the videos. On the
other hand, the observers were blinded to the clinical history and the pathologic
diagnosis and to each other’s evaluation. No prior selection was made based on the
quality of recorded images to avoid inducing any bias in the IOA evaluation.
Video sequence of contrast-enhanced harmonic endoscopic ultrasonography (left panel).
The video sequence also included a B-mode standard endoscopic ultrasound image of
the lesion of interest (right panel).
Selection of observers and evaluation of videos
A total of 11 endoscopists from six EUS centers (Santiago de Compostela, Spain; Braga,
Portugal; Porto, Portugal; Viana do Castelo, Portugal; Guimarães, Portugal; Gothenburg,
Sweden) participated in this study. They were divided into four groups according to
their experience in EUS and CH-EUS.
Group A included two endosonographers (J. I.-G. and J. L.-N) with long experience
in EUS (> 1000 procedures) and CH- EUS (> 200 procedures).
Group B included three endosonographers (B. L., L. L., and J.-B. S.) with 3 months
of experience in EUS (> 100 procedures) and CH-EUS (> 20 procedures).
Group C included three endosonographers (P. M., P. P.-N., and P. B.) with long experience
in EUS (> 1000 procedures) but no experience in CH-EUS.
Group D included three endoscopists (A. F., A. C. C., and B. G.) with no experience
in EUS or CH- EUS.
A kickoff session of 20 minutes was undertaken to share the principles of the techniques
and to make everybody acquainted with the parameters of CH-EUS under evaluation.
Observers were asked to classify the lesion of interest, based on its overall degree
of enhancement in comparison with the surrounding structures, as one of three types:
1, hypo-enhancement; 2, iso-enhancement; 3, hyper-enhancement. The degree of enhancement
was evaluated during the arterial phase starting from the first arrival of contrast
(usually in 10 – 20 seconds) until approximately 30 to 45 seconds [17].
We also calculated the diagnostic accuracy for pancreatic adenocarcinoma by comparing
the pattern of CH-EUS indicative of pancreatic adenocarcinoma with the final diagnosis
as previously described. We considered a hypo-enhanced contrast pattern in CH-EUS
as indicative of pancreatic adenocarcinoma.
Statistical analysis
The Fleiss kappa (κ) statistic was used to evaluate the IOA among observers. An individual κ value for each group of observers, as well as an overall κ value, was determined for CH-EUS. The κ values were interpreted according to the guidelines proposed by Landis and Koch [18]. The κ statistic allocates a score of 0 if the agreement is no better than would be expected
by chance, whereas perfect agreement is indicated by a κ value of 1. Scores can also be negative if there is consistent disagreement. In detail,
κ values of 0.00 to 0.19 represent slight agreement, 0.20 to 0.39 fair agreement, 0.40
to 0.59 moderate agreement, and 0.60 to 0.79 substantial agreement; a value of more
than 0.80 is considered almost perfect agreement. The κ values were considered statistically significant when the 95 %CI of the κ values was superior to 0. Bootstrap resampling was used to calculate the 95 %CI of
the κ values. Statistical comparison of κ values between groups was done with the κ analysis extension for ArcView 3.2. We also evaluated the sensitivity, specificity,
positive predictive value, negative predictive value, and AUROC of each group for
the final diagnosis of pancreatic adenocarcinoma by using the hypo-enhanced pattern
in CH-EUS as indicative of the presence of pancreatic adenocarcinoma [16].
With the exception of the comparison of κ values (see above), all statistical analyses were performed with SPSS 18.0 software
(Chicago, Illinois, USA). Differences with a P value of < 0.05 were considered significant.
Results
Patients’ characteristics
A total of 60 patients (17 women and 43 men with a mean age of 64 ± 15 years) were
included in the study ([Table 1]). The mean size of the pancreatic masses was 36.5 ± 15.9 mm. The lesions were located
mostly in the pancreatic head. The diagnosis was based on EUS-FNA in 43 patients,
on EUS-FNB (endoscopic ultrasonography-guided fine-needle biopsy) in 14 patients,
on surgery in 1 patient, and on follow-up in 2 patients. As determined according to
the reference methods, the final diagnoses were as follows: pancreatic adenocarcinoma
(45 patients), inflammatory mass in the context of chronic pancreatitis (10 patients),
pancreatic neuroendocrine tumor (3 patients), autoimmune pancreatitis (1 patient),
and metastatic colon cancer metastasis (1 patient).
Table 1
Characteristics of patients and pancreatic lesions included in an analysis of interobserver
agreement of contrast-enhanced harmonic endoscopic ultrasonography in the evaluation
of solid pancreatic lesions.
|
Age of patients, mean± SD, y
|
64 ± 15
|
|
Gender of patients (females : males)
|
17 : 43
|
|
Lesion size, mean ± SD, mm
|
36.5 ± 15.9
|
|
Location of lesions in pancreas (head/body/tail)
|
43/15/2
|
|
Final diagnosis
Pancreatic adenocarcinoma
Inflammatory mass in the context of chronic pancreatitis
Pancreatic neuroendocrine tumor
Autoimmune pancreatitis
Colon cancer metastasis
|
45
10
3
1
1
|
SD, standard deviation.
Interobserver agreement
The IOA evaluation data are presented in [Table 2]. The overall IOA for CH-EUS was fair. Group A had the highest IOA, followed by groups
C, B, and D. The IOA of groups A and C was significantly higher than that of group
D. No significant difference was seen between groups A, B, and C or between groups
B and D in terms of IOA.
Table 2
Results of interobserver agreement for each group of endosonographers.
|
Group (number of observers)
|
κ (95 %CI)
|
Interpretation of agreement
|
|
A (2 observers with long experience in EUS and CH-EUS)
|
0.63 (0.45 – 0.85)[1]
|
Substantial
|
|
B (3 observers with short experience in EUS and CH-EUS)
|
0.38 (0.22 – 0.55)
|
Fair
|
|
C (3 observers with long experience in EUS and no experience in CH-EUS)
|
0.54 (0.39 – 0.71)[1]
|
Moderate
|
|
D (3 observers with no experience in EUS or CH-EUS)
|
0.21 (0.07 – 0.36)
|
Fair
|
|
Overall (11 observers)
|
0.32 (0.22 – 0.41)
|
Fair
|
κ, kappa value; 95 %CI, 95 % confidence interval; EUS, endoscopic ultrasonography;
CH-EUS, contrast-enhanced harmonic endoscopic ultrasonography.
Data are shown as κ (95 %CI).
1
P < 0.05 vs. group D.
Diagnostic accuracy for pancreatic adenocarcinoma
The diagnostic accuracy data for pancreatic adenocarcinoma are presented in [Table 3]. Group A had the highest diagnostic accuracy, followed by groups C, B, and D. The
diagnostic accuracy of group A was not significantly different from that of group
C, but it was significantly higher than that of groups B and D. No significant difference
was seen between groups B, C, and D in terms of diagnostic accuracy.
Table 3
Diagnostic accuracy of each group of endosonographers for pancreatic adenocarcinoma.
|
Group A
|
Group B
|
Group C
|
Group D
|
Overall
|
|
Sensitivity (95 %CI)
|
79.6 (70.3 – 87.1)
|
61.9 (53.5 – 69.8)
|
64.0 (55.6 – 71.7)
|
58.5 (50.1 – 66.6)
|
64.6 (60.6 – 68.8)
|
|
Specificity (95 %CI)
|
54.6 (32.2 – 75.6)
|
48.5 (30.8 – 66.4)
|
51.5 (33.6 – 69.2)
|
42.4 (25.5 – 60.8)
|
48.8 (39.6 – 58.0)
|
|
PPV (95 %CI)
|
88.6 (80.1 – 94.4)
|
84.3 (76.0 – 90.6)
|
85.5 (77.5 – 91.4)
|
81.9 (73.2 – 88.7)
|
84.9 (81.1 – 88.2)
|
|
NPV (95 %CI)
|
37.5 (21.1 – 56.3)
|
22.2 (13.3 – 33.6)
|
24.3 (14.8 – 36.0)
|
18.7 (10.6 – 29.3)
|
23.7 (18.5 – 29.5)
|
|
AUROC (95 %CI)
|
0.67 (0.58 – 0.75)
|
0.55 (0.48 – 0.63)[1]
|
0.58 (0.50 – 0.65)
|
0.51 (0.43 – 0.58)[1]
|
0.57 (0.53 – 0.61)
|
95 %CI, 95 % confidence interval; PPV, positive predictive value; NPV, negative predictive
value; AUROC, area under the curve under summary receiver operating characteristic.
Results of the evaluation of 45 pancreatic adenocarcinoma and 15 non-pancreatic adenocarcinoma
lesions (inflammatory mass in the context of chronic pancreatitis, n = 10; neuroendocrine
tumor, n = 3; autoimmune pancreatitis, n = 1; metastasis, n = 1) were included for
analysis. Pancreatic adenocarcinoma was defined by the presence of a hypo-enhanced
contrast pattern in contrast-enhanced harmonic endoscopic ultrasonography (CH-EUS).
1
P < 0.05 vs. group A.
Discussion
This is the first study evaluating the IOA of CH-EUS in the evaluation of SPLs by
endoscopists with different levels of experience in EUS and CH-EUS. Our data suggest
that CH-EUS is reproducible in the evaluation of SPLs, even between endoscopists with
no or limited experience in EUS and/or CH-EUS. Our data also suggest that experience
in both EUS and CH-EUS influences the IOA. This finding is based on the observation
that group A (observers with long experience in EUS and CH-EUS) had the highest IOA
for CH-EUS (κ = 0.63; 95 %CI 0.45 – 0.85), whereas group D (observers with no experience in EUS
and CH-EUS) had the lowest IOA for CH-EUS (κ = 0.21; 95 %CI 0.07 – 0.36; P < 0.05). The fact that group B (observers with short experience in EUS and CH-EUS)
and group C (observers with long experience in EUS but no experience in CH-EUS) had
similar IOA for CH-EUS (κ = 0.38; 95 %CI 0.22 – 0.55 vs. κ = 0.54; 95 %CI 0.39 – 0.71; P > 0.05) suggests that long experience in EUS may influence the IOA of CH-EUS and
compensate for the lack of experience in CH-EUS.
Some studies have reported IOA in the evaluation of SPLs by CH-EUS, but only two studies
compared endosonographers with different levels of experience in EUS [19]
[20]. Fusaroli et al. evaluated the IOA of CH-EUS in 40 SPLs conducted by eight endosonographers
who were experienced in EUS and seven endosonographers who were not experienced in
EUS [19]. They reported a κ value for contrast uptake of 0.56 for all endosonographers, 0.56 for the experienced
endosonographers, and 0.55 for the endosonographers who were not experienced. Gincul
et al. evaluated the IOA of CH-EUS in 100 SPLs conducted by five senior and two junior
endosonographers [20], reporting a κ value of 0.66 for all endosonographers, 0.65 for the senior endosonographers, and
0.76 for the junior endosonographers. Kitano et al. evaluated the IOA of CH-EUS in
277 SPLs conducted by two endosonographers experienced in EUS and CH-EUS [21], reporting a κ value for contrast uptake of 0.95. Even between experienced endosonographers, these
studies show a large range of κ values (0.56 – 0.95) for the evaluation of contrast uptake by SPLs. In our study,
the κ value (0.63) for the evaluation of contrast uptake by SPLs by experienced endosonographers
fits within the range of previously reported κ values. On the other hand, contrary to the studies of Fusaroli et al. and Gincul et al.,
who reported no difference between endosonographers who were experienced and those
who were not experienced, our study found a difference between endosonographers with
different levels of experience in EUS and CH-EUS.
This is also the first study comparing the diagnostic accuracy of CH-EUS for pancreatic
adenocarcinoma by endoscopists with different levels of experience in EUS and CH-EUS. The
results of the diagnostic accuracy evaluation were very similar to those of the IOA
evaluation. As for IAO, our data suggest that experience in EUS or CH-EUS influences
the diagnostic accuracy. This is based on the observation that diagnostic accuracy
of CH-EUS was highest in group A (AUROC = 0.67; 95 %CI 0.58 – 0.75) and lowest in
group D (AUROC = 0.51; 95 %CI 0.43 – 0.58; P < 0.05). The fact that group B (observers with short experience in EUS and CH-EUS)
and group C (observers with long experience in EUS but no experience in CH-EUS) had
similar diagnostic accuracy for CH-EUS (AUROC = 0.55; 95 %CI 0.48 – 0.63 vs. AUROC = 0.58;
95 %CI 0.50 – 0.65; P > 0.05) suggests that long experience in EUS may influences the diagnostic accuracy
of CH-EUS and compensate for the lack of experience in CH-EUS.
Only one previous study has compared the diagnostic accuracy of CH-EUS for pancreatic
adenocarcinoma by observers with different levels of experience in EUS [20]. In this study, the observer accuracies for the diagnosis of pancreatic adenocarcinoma
by CH-EUS ranged from 81 % to 91 %, with no statistically significant difference found
between the senior and junior endosonographers [20]. However, in this study, no clear definition of senior and junior endosonographer
is provided. Thus, the difference in the experience of the senior and junior endosonographers
is not clear. This would be important to compare our data with the data of that study.
Previous long experience in EUS seems to have a significant influence on both the
IOA and diagnostic accuracy of CH-EUS. This is based on the observation that the IOA
and diagnostic accuracy of CH-EUS in group C were similar to those of groups A and
B, even though group C had no experience in CH-EUS.
A limitation of CH-EUS is that in the qualitative image analysis performed, the findings
are amenable to subjective interpretation. Thus, methods for the quantitative assessment
of CH-EUS have been recently developed, such as the contrast uptake ratio index and
the time–intensity curve [16]
[22]
[23]. Although these methods have already proved to be helpful, there are still some
limitations. All are based on the computed automated analysis of regions of interest
that are selected subjectively, thus allowing the generation of selection bias. Moreover,
they have not yet been proved to be superior to the qualitative analysis of CH-EUS
[16]
[22]
[23].
This study has some weaknesses. First, the small sample size, the low number of observers
per group, and the low rate of non-neoplastic lesions (although similar to that of
clinical practice) may have influenced our data. In fact, a major limitation of the
study is the smaller number of observers in group A (two observers) than in the other
groups (three observers per group). This was the result of the low number of available
experts in CH-EUS and may have led to overestimation of the IOA of group A, compromising
the comparison between this group and the other groups. Even so, this does not compromise
either the overall data for the IOA of CH-EUS or the comparison of the other three
groups, and thus does not influence the major conclusions of the study. Second, the
procedures were done by the same experts who later evaluated the lesions (group A).
To reduce this bias, each video sequence was labeled with a random number, and the
experts blindly evaluated the lesions. Third, each video sequence also included a
B-mode standard EUS image. Although this is similar to the clinical practice of CH-EUS,
the evaluation of a B-mode standard EUS image could influence the evaluation of CH-EUS
images, thus possibly contributing to overestimation of the IOA and diagnostic accuracy
of observers with long experience in EUS (groups A and C). It has been shown that
B-mode standard EUS may show changes strongly consistent with malignant tumor in 30 %
of SPLs [24]. Finally, as we did not assess the intra-observer agreement of CH-EUS, we cannot
make any conclusions about intra-observer reproducibility of CH-EUS.
The IOA and diagnostic accuracy of CH-EUS can be influenced by the contrast agent
that is used. Besides SonoVue (phospholipid-stabilized microbubbles of sulfur hexafluoride),
which was used in our study, Sonazoid (microbubbles of perfluorobutane with a lipid
membrane encapsulation; Daiichi Pharmaceutical, Tokyo, Japan, and GE Healthcare, Chalfont
St. Giles, United Kingdom) is also a commonly used ultrasound contrast agents for
CH-EUS [13]
[25]. It has been shown that the two ultrasound contrast agents differ in terms of intensity
and duration of signaling after infusion. CH-EUS images obtained with SonoVue disappear
within 60 seconds, which limits the duration of observation. In contrast, after infusion
of Sonazoid, parenchymal perfusion can be observed throughout the pancreas for at
least 90 seconds. Thus, the longer-lasting effect of Sonazoid may improve the observation
of the pancreas by CH-EUS. Nonetheless, no study has directly compared the two contrast
agents in the evaluation of SPLs. Moreover, in our study, we evaluated only the arterial
phase, starting from the first arrival of contrast (usually in 10 – 20 seconds) until
approximately 30 to 45 seconds, so it is very unlikely that our results would be different
with Sonazoid.
In conclusion, we present the first multicentric study of the IOA and diagnostic accuracy
of CH-EUS in the evaluation of SPLs by endoscopists with different levels of experience
in EUS and CH-EUS. Our data suggest that CH-EUS is reproducible in the evaluation
of SPLs, even in groups with no or limited experience in EUS and/or CH-EUS. The study
also hints that long experience in EUS is a major contributor to the IOA and diagnostic
accuracy of CH-EUS. Nonetheless, these data should undergo external validation in
larger studies.
