Explanations of Statistical Concepts: Can they Penetrate the Haze of Bayes?

Summary

This experiment tested the ability of logical explanations to promote the use of statistical concepts. Sixty physicians in four experimental groups, matched for level of medical statistical training, were given a questionnaire that assessed their ability to rank the diagnosticity of various diagnostic test results. Each question was designed to test physician understanding of the impact of a particular statistical parameter, e.g., sensitivity and specificity, test-retest reliability. Logical explanations of the sources of test error increased the percentage of answers judged to be correct on the basis of published data. This was confirmed for a variety of questions. They tended to be less useful, however, when physicians expressed misinterpretations of statistical concepts or test patterns were not correctly perceived. The implications of these findings for medical education and for the design of computer-based explanation systems are clarified.

In diesem Experiment wurde die Fähigkeit logischer Erklärungen zur Förderung der Anwendung statistischer Begriffe untersucht. Sechzig Ärzten mit gleichem Niveau medizinisch-statistischer Ausbildung in vier Studiengruppen wurde ein Fragebogen gegeben, der ihre Fähigkeit zur Wertung der diagnostischen Macht verschiedener diagnostischer Testergebnisse einschätzen sollte. Jede Frage war so formuliert, daß sich aus der Antwort ergeben sollte, inwieweit der Arzt die Wirkung eines bestimmten statistischen Parameters (z.B. Sensitivität und Spezifizität, Test-Retest-Zuverlässigkeit) verstehen konnte. Logische Erklärungen der Quellen von Testfehlern erhöhten den Prozentsatz der aufgrund veröffentlichter Daten für richtig erachteten Antworten. Dies wurde für eine Vielfalt von Fragen bestätigt. Sie erwiesen sich jedoch als weniger nützlich, wenn Ärzte Fehlinterpretationen statistischer Begriffe zum Ausdruck brachten oder wenn Testmuster nicht richtig erkannt wurden. Die Folgerungen dieser Feststellungen für die ärztliche Ausbildung und für die Planung computerunterstützter Erklärungssysteme werden geklärt.

Key-Words:

Diagnostic Tests - Decision-making - Psychological Studies of Clinical Judgement - Computer-based Consultation Systems - Artificial Intelligence

Schlüssel-Wörter:

Diagnostische Tests - Entscheidungsfindung - psychologische Studien über klinische Beurteilung - computerunterstützte Konsultationssysteme - künstliche Intelligenz

^* Dr. Politser is recipient of Research Career Development Award LM 00080 from the National Library of Medicine. This research also was supported in part by Grants LM 04132, LM 04086 and LM 03366 from the National Library of Medicine 3.nd Grant HS 04726 from the National Center for Health Services Research.

• References

• 1 Ajzen I.. Intuitive Theories of Events and the Effects of Base-Rate Information on Prediction. J. Person, soc. Psych 1977; 35: 303-314.
• 2 Bar-Hillel M.. The Base Rate Fallacy in Probability Judgements. Acta Psychol 1980; 44: 211-233.
• 3 Berwick D. M., Fineberg H. V., Weinstein M. D.. When Doctors Meet Numbers. Presented at the Second Annual Meeting of the Society for Medical Decision Making, Washington, D. C 1980
• 4 Bleich H. L.. Computer-based Consultation. Electrolyte and Acid-base Disorders. Amer. J. Med 1972; 53: 285-291.
• 5 Bourne Jr. L. E., Guy D. E.. Learning Conceptual Rules. II. The Role of Positive and Negative Instances. J. exp. Psych 1968; 77: 488-494.
• 6 Casscells W., Schoenberger A., Gray-boys T. B.. Interpretation by Physicians of Clinical Laboratory Results. New Engl. J. Med 1978; 299: 999-1001.
• 7 Chapman L. J., Chapman J. P.. Illusory Correlation as an Obstacle to the Use of Valid Psychodiagnostic Signs. J. abnorm. Psych 1969; 74: 271-280.
• 8 Chase W. G., Simon H. A.. Perception in Chess. Cogn. Psychol 1973; 4: 55-61.
• 9 Christensen-Szalanski J. J., Bushyhead J. B.. Physicians’ Misunderstanding of Normal Findings. Med. Decis. Mak 1983; 3: 169-175.
• 10 Detmer D. E., Fryback D. G., Gassner K.. Heuristics and Biases in Medical Decision Making. J. med. Educ 1978; 53: 682-683.
• 11 Eddy D. M.. Screening for Cancer: Theory, Analysis and Design. Englewood Cliffs, N. J.: Prentice-Hall; 1980
• 12 Einhorn H. J., Kleinmuntz D. N., Klein-muntz B.. Linear Regression and Process-tracing Models of Judgment. Psychol. Rev 1979; 86: 465-485.
• 13 Erdman H.. A Computer Consultation Program to Help Non-psychiatric Physicians Choose Treatments for Depression. Ph. D. Dissertation, Dept. of Industrial Engineering, University of Wisconsin
• 14 Feinstein A. R.. The Haze of Bayes, the Aerial Palaces of Decision Analysis, and the Computerized Ouija Board. Clin. Biostat 1976; 21: 482-496.
• 15 Golding S. L., Rorer L. G.. Illusory Correlation and Subjective Judgement. J. abnorm. Psych 1972; 80: 249-260.
• 16 Gorry G., Kassirer J., Essig A., Schwartz W.. Decision Analysis as the Basis for Comper-aided Management of Acute Renal Failure. Amer. J. Med 1973; 55: 473-484.
• 17 Greenland P., Muschlin A. I., Griner P. F. Discrepancies between Knowledge and Use of Diagnostic Studies in Asymptomatic Patients. J. med. Educ 1979; 54: 863-869.
• 18 Griner P. F, Mayewski R. J., Mushline A. I., Greenland P.. Selection and Interpretation of Diagnostic Tests and Procedures. Ann. int. Med 1981; 94: 553-600.
• 19 Hovland C. I., Weiss W.. Transmission of Information Concerning Concepts through Positive and Negative Instances. J. exp. Psych 1953; 45: 175-182.
• 20 Jacquez J. A., Norusis M. J.. The Importance of Symptom Non-independence. In DeDombal F T.. and Gremy F.. (Eds) Diagnosis, Decision Making and Medical Care. Amsterdam: North-Holland Publ. Co; 1976
• 21 Kahneman D., Tversky A.. Judgement Under Uncertainty: Heuristics and Biases. Science 1974; 184: 1124-1131.
• 22 Komaroff A. L.. Algorithms and the "Art" of Medicine. Amer. J. Public Hlth 1982; 72: 10-12.
• 23 Lopes L.. Doing the Impossible: A Note on Induction and the Experiences of Randomness. Department of Psychology, University of Wisconsin 1980
• 24 Ludwig D., Heilbronn D.. The Design and Testing of a New Approach to Computer-aided Differential Diagnosis. Meth. Inform. Med 1983; 22: 156-166.
• 25 Politser P. E.. Decision Analysis and Clinical Judgement: A Re-evaluation. Med. Dec. Mak 1982; 1: 368-389.
• 26 Politser P. E.. The Evaluation of Repeated Medical Tests: Logical and Statistical Considerations. Ph.D. Dissertation, The University of Michigan 1982
• 27 Politser P. E.. Reliability, Decision Rules, and the Value of Repeated Test. Med. Dec. Mak 1982; 2: 47-69.
• 28 Slovic P., Lichtenstein S.. Comparison of Bayesian and Regression Approaches to the Study of Information Processing in Judgement. Org. Behav. Hum. Perform 1971; 6: 649-744.
• 29 Smedslund J.. The Concept of Correlation. Scand. J. Psych 1963; 4: 165-174.
• 30 Swensson R. G.. A Two-stage Detection Model Applied to Skilled Visual Search by Radiologists. Percept. Psychophys 1080; 27: 11-16.
• 31 Thorngate W.. Efficient Decision Heuristics. Behav. Sci 1980; 25: 219-225.
• 32 Wagenaar W. A.. Appreciation of Conditional Probabilities in Binary Sequences. Acta Psychol 1970; 34: 348-356.
• 33 Wallis J. W., Shortliffe E. H.. Explanatory Power for Medical Expert Systems: Studies in the Representation of Causal Relationships for Clinical Consultations. Meth. Inform. Med 1982; 21: 127-136.