Validity and Reliability of 3D US for the Detection of Erosions in Patients with Rheumatoid Arthritis Using MRI as the Gold Standard

Abstract

Purpose: To test the reliability and validity of a 3D US erosion score in RA using MRI as the gold standard.

Materials and Methods: RA patients were examined with 3D US and 3 T MRI over the 2^nd and 3^rd metacarpophalangeal joints. 3D blocks were evaluated by two investigators. The erosions were estimated according to a semi-quantitative score (SQS) (0 – 3) and a quantitative score (QS) (mm²). MRI was evaluated according to the RAMRIS score. For the estimation of reliability, intra-class correlation coefficients (ICC) were used. Validity was tested using Spearman’s rho (r_s). The sensitivity and specificity were also calculated.

Results: 28 patients with RA were included. The ICC for the inter-observer reliability in the QS was 0.41 and 0.13 for the metacarpal bone and phalangeal bone, respectively, and 0.86 and 0.16, respectively, in the SQS. The ICC for the intra-observer reliability in the QS was 0.75 and 0.48 for the metacarpal bone and phalangeal bone, respectively, and 0.83 and 0.60, respectively, in the SQS. The correlation with MRI for the metacarpal bone was significant, with values of 0.73 (p = 0.0001) (SQ) and 0.74 (p = 0.0001) (SQS). For the phalangeal bone, bad correlation was found: 0.28 (p = 0.145) (SQ) and 0.26 (p = 0.57) (SQS). The sensitivity and specificity for the metacarpal bone were 86 % and 85 % respectively. For the phalangeal bone they were 60 % and 97 %, respectively.

Conclusion: Good inter- and intra-observer reliability and correlation with MRI were seen in the assessment of erosions with 3D US in the metacarpal bone, while the results were low and insignificant for the phalangeal bone, indicating that 3D US still has room for improvement.

Zusammenfassung

Ziel: Bestimmung der Verlässlichkeit und Aussagekraft der 3-D-US-Erosionsskala bei RA unter Verwendung von MRI als Goldstandard.

Material und Methoden: RA-Patienten wurden mittels 3-D-US und 3T-MRT über dem 2. und 3. Metakarpophalangealgelenk untersucht. Die 3-D-Blöcke wurden von 2 Untersuchern bewertet. Die Erosionen wurden nach dem semiquantitativen Score (SQS) (0 – 3) und dem quantitativen Score (QS) (mm²) bewertet. Die MRT wurde mittels RAMRIS-Score beurteilt. Für die Bewertung der Verlässlichkeit wurden Intraklasse-Korrelationskoeffizienten (ICC) benutzt. Die Aussagekraft wurde mittels Spearmans Rho (rs) bestimmt. Die Sensitivität und Spezifität wurde ebenfalls bestimmt.

Ergebnisse: Es wurden 28 Patienten mit RA eingeschlossen. Die ICCs der Interobserver-Reliabitität beim QS betrugen für den metakarpalen Knochen 0,41 und für den phalangealen Knochen 0,13. Beim SQS betrugen diese 0,86 für den metakarpalen und 0,16 für den phalangealen Knochen. Die ICCs der Intraobserver-Reliabitität beim QS betrugen für den metakarpalen Knochen 0,75 und für den phalangealen Knochen 0,48. Beim SQS betrugen diese 0,83 für den metakarpalen und 0,60 für den phalangealen Knochen. Die Übereinstimmung mit der MRT für den metakarpalen Knochen war signifikant mit Werten von 0,73 (p = 0,0001) für den SQ und 0,74 (p = 0,0001) für den SQS. Beim phalangealen Knochen konnte mit 0,28 (p = 0,145) für den SQ und 0,26 (p = 0,57) für den SQS keine Übereinstimmung gefunden werden. Beim metakarpalen Knochen betrug die Sensitivität 86 % und die Spezifität 85 %. Beim phalangealen Knochen betrug die Sensitivität 60 % und die Spezifität 97 %.

Schlussfolgerung: Bei der Bewertung der Erosionen des metakarpalen Knochens mittels 3-D-US wurden eine gute Inter- und Intraobserver-Reliabilität sowie eine gute Übereinstimmung mit der MRT beobachtet. Gleichzeitig waren die Ergebnisse für den phalangealen Knochen schlecht und nicht signifikant, was darauf hinweist, dass der 3-D-US noch verbesserungsfähig ist.

Key words

hand - 3 D ultrasound - rheumatoid arthritis - erosions

Introduction

The use of ultrasound (US) in rheumatology has increased rapidly within the last decade. Until recently, the only US modality available was 2D. 2D US has been shown to provide a more sensitive measurement than X-ray for the detection of erosion in patients with rheumatoid arthritis (RA) [1] [2]. When erosions detected on 2D US are compared to erosions detected on CT or MRI, the correlations are acceptable [2] [3] [4].

Recently, 3D US has emerged as an application in some high-end US machines. With the introduction of high-frequency volumetric probes, the use of 3D US as a part of both rheumatological practice and research may be realistic in the near future [5]. In 3D US examinations, multiple 2D US images form a 3D block. Subsequently, it is possible to scroll through the 3D block in any given plane and evaluate multiple planes simultaneously.

US is perceived to be an operator-dependent procedure [6] [7]. This disadvantage caused by various operators may partly be solved by using 3D US, because the variation in image acquisition is minimized [6].

The US task force in OMERACT (Outcome Measures in Rheumatology) defines erosion as an irregularity of the bone surface visible in two perpendicular planes [8]. Because it is possible in the post-processing of the 3D US examination to inspect two perpendicular planes of the same point simultaneously, 3D US could be assumed to be a reliable and valid examination procedure in the assessment of erosions in patients with RA.

Only a few pilot studies of 3D US for the detection and evaluation of erosive changes in patients with RA have been published [5] [6] [9]. Therefore, further validation of 3D US is needed.

The main aim of this study was to test the inter- and intra-observer reliability of an erosion score in 3D US in RA MCP joints, and secondly, to test the concurrent validity of the erosion score using MRI as a reference method. In addition the sensitivity and specificity were evaluated by verifying the agreement regarding the presence and absence of erosions in comparison to MRI.

Materials and methods

Patients

28 patients with RA as defined by the American College of Rheumatology criteria [10] were enrolled in the study. The study population was a randomly selected part of a bigger cohort of RA patients who were examined with MRI to evaluate the progression of their disease. The reason for choosing MRI and not CT was the radiation risk for the patient using CT. The study was carried out in an outpatient clinic. In all patients both a 3D US examination and a 3 Tesla MRI of the 2^nd and 3^rd MCP joint were performed. The two examinations were performed on the same day. A clinical examination was also carried out.

Ultrasound

The US examinations were performed by a person with extensive musculoskeletal US experience (KE). The patients were seated opposite the investigator with the hand and forearm resting on an examination couch. The examinations were carried out using a Logic E9 with a 6 – 16-D volumetric probe (General Electrics Medical System, Milwaukee, WI). In all patients a 3D sweep (29^O) over the 2^nd and the 3^rd MCP joints was made on both the dorsal and palmar side. Additionally, a sweep of the radial part of the 2^nd MCP joint was performed. The image acquisitions were performed in the longitudinal plane and the probe was placed centrally over the joint space. The metacarpal and proximal phalangeal bones served as landmarks in all positions.

MRI

The MCP joints and wrist where examined in a 3 T Siemens Verio^® MR scanner with the patients supine and the hand along the side of the body using a semiflex 15-channel body coil and the following protocol: Gradient echo scout, coronal T1-weighted (T1 W) turbo spin echo (TSE), coronal and axial STIR and gradient echo 3D T1w fat-saturated VIBE before and after intravenous injection of single dose gadolinium contrast (Prohance, Bracco Italy, 0.1 mmol/kg).

Image evaluation US

In this study we defined erosions according to the US group in OMERACT: “An intraarticular discontinuity of bone surface that is visible in 2 perpendicular planes” [8]. The two persons performing the US evaluations were unaware of the results of the MRI evaluations.

The size of the erosions was measured both on a semi-quantitative scale (0 – 3) and a quantitative scale (mm²). In order to identify all erosions in two perpendicular planes, the same area of the bone in both the transverse and longitudinal planes was identified by scrolling in two planes simultaneously ([Figure 1]). In the quantitative score, an estimate of the size of the erosions in mm² was made by multiplying the obtained maximal width (transverse scan plane) and length (longitudinal scan plane).

Fig. 1 The left image shows the longitudinal plane. The right image shows the constructed transverse plane. The coronal plane is not shown. a Dorsal-central position MCP2. b Radial position MCP2.

Abb. 1 Das linke Bild zeigt die Längsebene. Das rechte Bild zeigt die konstruierte Transversalebene. Die koronale Ebene ist nicht gezeigt. a Dorsal-zentrale Position MCP2. b Radiale Position MCP2.

The semi-quantitative scale used was developed in cooperation with the US unit at Chapel Alleton Hospital in Leeds. The inter-observer reliability of the system was found to be very good (unpublished data). In the system the scores were defined as follows: 0 = no erosions, 1 = erosions covering less than one-third of the surface of the bone, 2 = erosions between one-third and two-thirds of the bone surface, 3 = erosions covering more than two-thirds of the bone surface. The semi-quantitative score was made in the longitudinal image.

Image evaluation MRI

The MRI evaluation was made by a third person who was unaware of the results of the US evaluations.

The coronal T1 TSE, and the 3D coronal T1w gradient echo VIBE pre- and post-contrast images were used for RAMRIS erosion scoring of the MCP joints [11].

Reliability and validity

The inter- and intra-observer reliability for the detection of erosions was investigated. In order to test the intra-observer reliability, one person (KE) evaluated all examinations twice after a period of at least 24 hours. To test the inter-observer reliability, fourteen of the examinations were evaluated by a second person well trained in US (UMD). Before the evaluation, a consensus was acquired between KE and UMD in the identification and scoring of erosions

In both joints a sum score for each of the two articular surfaces was calculated for both the semi-quantitative and the quantitative scores. The reliability was calculated by comparison of the sum scores of the various articular surfaces in the two joints.

Absolute agreement (presence/absence of erosions) was also calculated for the two bone surfaces.

The concurrent validity was investigated using MRI as the gold standard. The first US evaluation performed was used to calculate the correlation between 3D US and MRI. The correlation was calculated for each articular surface of the joint. Also the sensitivity and specificity in comparison to MRI were calculated.

Statistics

In this study intra- and inter-observer reliability was assessed in order to evaluate the agreement within and between ultrasound specialists. In addition, the absolute intra- and inter-observer agreement was calculated.

The relative reliability was assessed using Intraclass Correlation Coefficients (ICC) (2.1) that were calculated after testing for any systematic differences using t-tests. Qualitative descriptions of the ICC were done according to Fleiss JL [12], where an ICC > 0.75 is excellent; 0.40 > ICC > 0.75 indicates fair to good reliability and ICC < 0.40 indicates poor reliability. The ICC reflects the relative reliability as it evaluates the individual’s position within a group. 

The absolute reliability was calculated as the within-subject standard deviation calculated as the square root of the mean square error term (EMS) obtained from a two-way analysis of variance (ANOVA). This is called the measurement error (ME) and is unaffected by the range of the measured values. The ME is in the original data units.

The absolute agreement (presence/absence of erosions) was also calculated.

The agreement between US and MRI was assessed using Spearman’s correlation (r_s). The sensitivity and specificity in relation to MRI were also calculated.

Results

Patient characteristics

28 outpatients with RA with hand involvement were included in the study. Demographic data of the 28 patients are shown in [Table 1].

Table 1 Demographic data of the 28 RA patients, median (range).[1]

variable	total sample (n = 28)
sex, female/male	21/7 (75 %)
age	53 (21 – 84)
duration of RA	9 (1 – 20)
TJC	6 (0 – 14)
SJC	4 (0 – 12)
VAS	26 (7 – 70)
CRP	4.5 (0 – 41)
DAS28	4.03 (2.60 – 5.59)

¹ CRP = C-reactive protein; TJC = tender joint count; SJC = swollen joint count; VAS = visual analog scale – mm; DAS28 = 28-joint Disease Activity Score.

Reliability, validity, sensitivity and specificity

The relative and absolute inter- and intra-observer reliability was tested for both scoring systems ([Table 2]).

Table 2 Inter- and intra-observer reliability and absolute agreement for scoring of erosions in 3 D US.[1]

	ICC(2.1) (95 % CI)	within subject variation cm2 absolute variation	absolute agreement percent
inter-observer; QS; META n = 14	0.41 (0.46 – 0.68)	0.519	84 %
inter-observer; QS; PHA n = 14	0.13 (–0.22 – 0.46)	0.054	82 %
inter-observer; SQS; META n = 14	0.86 (0.76 – 0.93)
inter-observer; SQS; PHA n = 14	0.16 (–0.18 – 0.49)
intra-observer; QS; META n = 28	0.75 (0.63 – 0.86)	0.170	89 %
intra-observer; QS; PHA n = 28	0.48 (0.25 – 0.66)	0.031	93 %
intra-observer; SQS; META n = 28	0.83 (0.73 – 0.90)
intra-observer; SQS; PHA	0.60 (0.39 – 0.74)

¹ QS = Quantitative score; S-QS = Semi-quantitative score; META = Metacarpal bone; PHA = Phalangeal bone

The ICC (2.1) in the inter-observer test was fair for the metacarpal bone but poor for the phalangeal bone in the quantitative score. In the semi-quantitative score the ICC was considerably better for the metacarpal bone with excellent reproducibility. However, no difference between the two scoring systems was seen in the inter-observer reliability in the evaluation of the phalangeal bone.

The intra-observer reliability was better in all scores except for the semi-quantitative score of the metacarpal bone. In the intra-reliability test the results were also much better for the metacarpal bone than for the phalangeal bone. However, the within subject variation was much better in the evaluation of the phalangeal bone in both inter- and intra-observer reliability ([Table 2]).

The absolute intra- and inter-observer agreement was comparable for the metacarpal and phalangeal bone (presence or absences of erosions). In general the intra-observer absolute agreement was slightly higher than the inter-observer agreement ([Table 2]).

In comparison to the reliability testing, the correlation between both the quantitative and the semiquantitative scores between the metacarpal bone and the RAMRIS MRI score was acceptable and statistically significant. However, for the phalangeal bone the correlation for both scoring systems was poor and insignificant ([Table 3]).

Table 3 Correlation between erosion scores in 3 D US and MRI.

US vs. MRI	metacarpal bone	phalangeal bone
quantitative score	0.73 (p = 0.0001)	0.28 (p = 0.145)
semi-quantitative score	0.74 (p = 0.0001)	0.26 (p = 0.57)

Using the MRI RAMRIS score as a reference, the sensitivity for US for the detection of erosions for the metacarpal and phalangeal bones was 86 % and 60 %, respectively. The specificity was 85 % for the metacarpal bone and 97 % for the phalangeal bone.

Discussion

To the best of our knowledge, this is the first study investigating the reliability and validity of 3D US score for evaluating erosions in the MCP joints in patients with RA. The results showed good to excellent reliability in all tests in the assessment of the metacarpal bone. When assessing the phalangeal bone, the intra-observer reliability was only fair to good and the inter-observer reliability was poor. This might partly be explained by the difference in shape of the 2 bones – the phalangeal bone being more irregular. We believe that it is easier to recognize a change in the shape of the head of the metacarpal bone than a change in the base of the phalanx.

In the evaluation of the absolute variation it is interesting that, in contrast to the other data, these results are better for the phalangeal bone than the metacarpal bone. This may be due to the fact that erosions on the phalangeal bone in general are smaller, making the opportunity for variation limited.

With regard to the absolute variation, the difference between the two bones was not seen, which is understandable as the absolute agreement only investigates the agreement of either the presence or absence of erosions.

The same tendency as in the reliability testing was seen in the test of the concurrent validity. Here a good correlation was seen between the MRI RAMRIS score and 3D US in the assessment of the metacarpal bone, which was not the case in the assessment of the reliability in the phalangeal bone.

The ability of 3D US to detect erosions in the MCP joints has also been demonstrated in a study where 3D US was assessed using CT as the gold standard. In this study the specificity and sensitivity of 3D US were 84.1 % and 91.4 %, respectively [9]. These results are not directly comparable to ours as these results were not calculated for the various bone surfaces in the joint.

In a study by Finzel et al. [3], the correlation between 2D US and CT was investigated. In this study the bone surfaces were evaluated separately and, in contrast to our study, both the dorsal and volar parts of the bone were evaluated. In general, a good correlation, sensitivity and specificity were seen for the US examination, but the values were a little poorer than in our assessment of the metacarpal bone. However, the correlation in the study by Finzel et al. was poorest in the evaluation of the volar aspect of the phalangeal bone. Considering that we did not evaluate this part of the joint, we find that the results could be comparable.

In another study the inter-observer reliability in 2 RA patients examined with 3D was investigated [6]. In this study 12 rheumatologists evaluated the images using a semi-quantitative scoring system. They found a moderate Kappa value of 0.47, which is defined as only a fair reliability [13]. Thus, their reliability was considerably poorer than ours in the assessment of the metacarpal bone. In this study they did not distinguish between the various bones which again makes the comparison difficult.

Our inter-observer reliability of the metacarpal bone using semi-quantitative scoring is moderately higher than in a study testing the inter-observer reliability of MRI for the detection of erosions in MCP joints, in which they found an ICC of 0.58 [14].

Ultrasound has some obvious advantages compared to MRI with regard to both the accessibility and the comfort for the patient. The 3D US examination procedure is more comparable to MRI than 2D US, as it is possible to scroll through the different images in a block and to visualize two perpendicular plans at the same time ([Fig. 1]). Furthermore, it is possible, in accordance with MRI, to evaluate a whole anatomic area at any time, while in 2D US you only have the opportunity to evaluate the images stored during the examination. These advantages and the comparable reliability of 3D US with MRI in the metacarpal bone might suggest that 3D US in some cases can replace MRI in the monitoring of patients.

Theoretically, it is possible with 3D US to follow single erosions over time – this is an inherent advantage of 3D US. This has, however, not been investigated in any studies. Future longitudinal studies are therefore needed to establish whether 3D ultrasound has the capability to monitor the size of erosions over time.

In conclusion, good intra-observer and inter-observer reliability and good correlation with MRI in the assessment of erosions of the metacarpal head with 3D US were found. When evaluating the phalangeal bone, good reliability and correlation with MRI were not demonstrated. Thus, 3D US may be applied in the evaluation of erosions of the metacarpal bone. Future studies are needed to attempt to develop reliable and valid 3D scoring for erosions of the phalangeal bone and to investigate whether 3D US has the ability to monitor erosions over time.

Acknowledgments

This study was supported by the Danish Rheumatism Association and the Oak Foundation.

• Reference List

• 1 Wakefield RJ, Gibbon WW, Conaghan PG et al. The value of sonography in the detection of bone erosions in patients with rheumatoid arthritis: a comparison with conventional radiography. Arthritis Rheum 2000; 43: 2762-2770
• 2 Szkudlarek M, Klarlund M, Narvestad E et al. Ultrasonography of the metacarpophalangeal and proximal interphalangeal joints in rheumatoid arthritis: a comparison with magnetic resonance imaging, conventional radiography and clinical examination. Arthritis Res Ther 2006; 8: R52
• 3 Finzel S, Ohrndorf S, Englbrecht M et al. A detailed comparative study of high-resolution ultrasound and micro-computed tomography for detection of arthritic bone erosions 2. Arthritis Rheum 2011; 63: 1231-1236
• 4 Dohn UM, Ejbjerg BJ, Court-Payen M et al. Are bone erosions detected by magnetic resonance imaging and ultrasonography true erosions? A comparison with computed tomography in rheumatoid arthritis metacarpophalangeal joints. Arthritis Res Ther 2006; 8: R110
• 5 Filippucci E, Meenagh G, Delle SA et al. Ultrasound imaging for the rheumatologist. XX. Sonographic assessment of hand and wrist joint involvement in rheumatoid arthritis: comparison between two- and three-dimensional ultrasonography. Clin Exp Rheumatol 2009; 27: 197-200
• 6 Naredo E, Moller I, Acebes C et al. Three-dimensional volumetric ultrasonography. Does it improve reliabililty of musculoskeletal ultrasound?. Clin Exp Rheumatol 2010; 28: 79-82
• 7 Albrecht K, Muller-Ladner U. Quantification of the synovial perfusion in rheumatoid arthritis using Doppler ultrasonography. Clin Exp Rheumatol 2007; 25: 630-638
• 8 Wakefield RJ, Balint PV, Szkudlarek M et al. Musculoskeletal ultrasound including definitions for ultrasonographic pathology. J Rheumatol 2005; 32: 2485-2487
• 9 Peluso G, Bosello SL, Michelutti A et al. Detection of bone erosions in erly rheumatoid arthritis: 3D Ultrasonography compared with computed tomography shows ahigh specificity and sensitivity for MCP joints. ARD 2011; [Supplement 3], FRI0092. 1-6-2011. Ref Type: Abstract
• 10 Arnett FC, Edworthy SM, Bloch DA et al. The American Rheumatism Association 1987 revised criteria for the classification of rheumatoid arthritis. Arthritis Rheum 1988; 3: 315-324
• 11 Ostergaard M, Peterfy C, Conaghan P et al. OMERACT Rheumatoid Arthritis Magnetic Resonance Imaging Studies. Core set of MRI acquisitions, joint pathology definitions, and the OMERACT RA-MRI scoring system 6. J Rheumatol 2003; 30: 1385-1386
• 12 Fleiss JL. The design and analysis of clinical experiments. New York: Wiley; 1986
• 13 Landis JR, Koch GG. The measurements of observer agreement for categorical data. Biometrics 1977; 33: 159-174
• 14 Lassere M, McQueen F, Ostergaard M et al. OMERACT Rheumatoid Arthritis Magnetic Resonance Imaging Studies. Exercise 3: an international multicenter reliability study using the RA-MRI Score 21. J Rheumatol 2003; 30: 1366-1375