Interrater Variability in Clinical Studies – A study on Pulmonary Artery Embolism (PE) Diagnosis with V/P-SPECT/CT

Ziel/Aim This study aims to determine the inter-rater reliability between two raters using Cohen’s kappa (k) to investigate a reliable optical medical diagnosis within the scope of a semi-automatic evaluation.

Methodik/Methods Retrospectively, n = 200 patients were conventionally diagnosed by V/P-SPECT/ldCT by two raters. For rater 1, a binary dichotomous grading was given (score 0: no PE; score 4: clear PE). For rater 2, a five-level weighted score was given (score 0: no PE, score 1: probably no PE; score 2: equivocal, score 3: probably PE; score 4: clear PE), which was converted into a binary scoring by setting the PE threshold to either score 2, 3, or 4. To investigate the common agreement, Cohen’s kappa evaluation was performed for different categorizations of the entire lung as well as for the five individual lung lobes. Additionally, the confidence interval (CI: 95 %) as well as the sensitivity (Se) and specificity (Sp) of both raters were determined and evaluated. Se and Sp were calculated to provide more information about the diagnosis robustness in case of unequal distributions of the contingency tables.

Ergebnisse/Results The largest agreement between the raters was found with a PE ratio of 25 % (n = 49) for rater 1 and 27 % (n = 54) for rater 2. An entire lung analysis resulted in k = 0.75 with CI (95 %): [0.68-0.82] with rater 1 (Se = 0.76, Sp = 0.98) and rater 2 (Se = 0.79, Sp = 0.97). A single lung lobe analysis resulted in k(RUL)=0.71, k(RML)=0.71, k(RLL)=0.86, k(LUL)=0.74, k(LLL)=0.73.

Schlussfolgerungen/Conclusions The results were not affected by the method of diagnosis classification. There was no significant difference between the analysis of total and single lobes. Sensitivity and specificity are not significantly different between the two raters. The numerically quantified evaluation revealed that the raters performed the classification according to PE size criteria.

Publication History

Article published online:
08 April 2021