Validation of the AI/RHEUM Knowledge Base with Data from Consecutive Rheumatological Outpatients

Abstract:

The diagnostic accuracy of AI/RHEUM, an experimental expert system for support in the diagnosis of rheumatic diseases, was assessed using a collection of data in a cohort of 1,570 consecutive outpatients of a Dutch rheumatological clinic. Computer diagnoses based on these data and diagnostic predictions made by rheumatologists were compared with reference diagnoses that had been obtained by consensus of rheumatologists after 6-12 months follow-up.

Performance of the tested version of the AI/RHEUM knowledge base is presented by various methods. Sensitivity varied between 29% and 100% for different rheumatological diseases. Average sensitivity and specificity for all 26 diagnoses present in the knowledge base were 67% and 98%, respectively. Performance according to the level of confidence indicated that 78% of the “definite”, 65% of the “probable”, and 33% of the “possible” conclusions made by AI/RHEUM were in agreement with the reference diagnoses. These results approximated the predictions made by rheumatologists after a single, initial examination. The system was less accurate than it had appeared in previous evaluation studies with complex clinical cases. The AI/RHEUM knowledge base needed refining to diagnose early rheumatic complaints. This study further illustrates the need for objective and informative parameters for expressing accuracy of diagnostic support systems.

Key-words

Computer-assisted Diagnosis - Evaluation - Rheumatology

• REFERENCES

• 1 Young FE. Validation of medical software: present policy of the Food and Drug Administration. Ann Int Med 1987; 106: 628-9.
• 2 Lundsgaarde HP. Evaluating Medical Expert Systems. Soc Sci Med 1987; 24: 805-19.
• 3 Wyatt J, Spiegelhalter DJ. Evaluating medical expert systems: what to test, and how?. Med Inform 1990; 15: 205-17.
• 4 Indurkhya N, Weiss S. Models for measuring performance of medical expert systems. Artif Intell Med 1989; 01: 61-70.
• 5 Mars NJI, Miller PL. Knowledge acquisition and verification tools for medical expert systems. Med Decis Making 1987; 07: 6-11.
• 6 Bernelot HJMoens, Van der Korst JK. Computer-assisted diagnosis of Rheumatic Disorders: a Review. Semin Arthritis Rheum 1991; 21: 156-69.
• 7 Kingsland LC, Lindberg DAB, Sharp GC. Anatomy of a knowledge-based consultant system: AI/RHEUM. Clinical Computing 1986; 03: 18-26.
• 8 Kingsland LC. The evaluation of medical expert systems: experience with the All RHEUM knowledge based consultant system in rheumatology. In: Proc 9th Annual Symposium on Computer Applications in Medical Care. Washington DC: IEEE Computer Society Press; 1985: 292-5.
• 9 Porter JF, Kingsland LC, Lindberg DAB. et al. The AI/RHEUM knowledge-based computer consultant system in rheumatology. Performance in the diagnosis of 59 connective tissue disease patients from Japan. Arthritis Rheum 1988; 31: 219-26.
• 10 Bernelot HJMoens, Van der Korst JK. Comparison of rheumatological diagnoses by a Bayesian program and by physicians. Meth Inf Med 1991; 30: 187-93.
• 11 Bernelot HJMoens, Van der Korst JK. Development and validation of a computer program using Bayes’ theorem to support diagnosis of rheumatic diseases. Ann Rheum Dis 1992; 51: 266-71.
• 12 De Dombal FT, Horrocks JC. Use of Receiver Operating Characteristic (ROC) curves to evaluate computer confidence threshold and clinical performance in the diagnosis of appendicitis. Meth Inf Med 1978; 17: 157-61.
• 13 Adlassnig K-P, Scheithauer W. Performance evaluation of medical expert systems using ROC curves. Comput Biomed Res 1989; 22: 297-313.
• 14 Bemmel JHvan, Willems JL. Standardization and validation of medical decision-support systems: the CSE project. Meth Inf Med 1990; 29: 261-2.
• 15 Bankowitz RA, McNeil MA, Challinor SM, Miller RA. Effect of a computer-assisted general medicine diagnostic consultation service on housestaff diagnostic strategy. Meth Inf Med 1989; 28: 352-6.
• 16 Kassirer JP. Our stubborn quest for diagnostic certainty. A cause of excessive testing. N Engl J Med 1989; 320: 1489-91.
• 17 Shortliffe EH. Computer programs to support clinical decision making. JAMA 1987; 258: 61-6.
• 18 Symmons DPM, Van Hemert AM, Vandenbroucke JP, Valkenburg HA. A longitudinal study of back pain and radiological changes in the lumbar spines of middle-aged women. II. Radiographic findings. Ann Rheum Dis 1991; 50: 162-6.
• 19 Kolarz G, Adlassnig K-P. Problems in establishing the medical expert systems CADIAG-1 and CADIAG-2 in rheumatology. J Med Systems 1986; 10: 395-405.
• 20 Schewe S, Herzer P, Kruger K. Prospective application of an expert system for the medical history of joint pain. Klinische Wochenschrift 1990; 68: 466-71.
• 21 Robertson WO. Quantifying the meanings of words. JAMA 1983; 249: 2621-2.
• 22 Bryant GD, Norman GR. Expression of probability: words and numbers. N Engl J Med 1980; 302: 411.