Methods Inf Med 1990; 29(04): 308-316
DOI: 10.1055/s-0038-1634794
ECG Interpretation Systems
Schattauer GmbH

Evaluation of ECG Interpretation Results Obtained by Computer and Cardiologists[*]

J. L. Willems
1   CSE Coordinating Center, Univ. of Leuven, Belgium
,
C. Abreu-Lima
2   Department of Cardiology, University of Porto, Portugal
,
P. Arnaud
3   INSEPvM, Unité 121, Hôp. Cardiovasculaire, Lyon, France
,
C. R. Brohet
4   Department of Cardiology, University of Louvain, Brussels, Belgium
,
B. Denis
5   Service de Cardiologie, Univ. of Grenoble, France
,
J. Gehring
6   Clinic for Cardiovascular Diseases, Höhenried, Federal Republic of Germany
,
I. Graham
7   Adelaide Hospital, Dublin, Ireland
,
G. van Herpen
8   University Hospital, Leiden, The Netherlands
,
H. Machado
9   University Hospital Santa Maria, Lisbon, Portugal
,
J. Michaelis
10   Institut für Medizinische Statistik und Dokumentation, Mainz, Federal Republic of Germany
,
S. D. Moulopoulos
11   Department of Therapeutics, University of Athens, Athens, Greece
› Institutsangaben
The research presented in this paper has been supported by the European Commission within the frame of its 2nd, 3rd and 4th Medical and Public Health Research Programmes of DG XII, Section Biomedical Engineering, and by various funding agencies in ten Member States of the European Community. The authors are grateful to Ms. I. Tassens for secretarial assistance and to Mr. I. Schoolmeesters for computer programming help.
Weitere Informationen

Publikationsverlauf

Publikationsdatum:
06. Februar 2018 (online)

Abstract

In an international project investigators from 25 institutes are trying to establish a common reference library and evaluation methods for testing the diagnostic performance of various ECG computer programs and of cardiologists, based on ECG-independent clinical information. A first set of 500 validated ECGs was collected and analyzed by fifteen different computer programs and nine cardiologists, seven of who analysed the ECG and five the VCG. A coding scheme was used to map individual diagnostic statements onto a common set. Combined program and referee results were obtained by weighted averaging. Preliminary results indicate that the classification accuracy of several programs can still be improved. However, it was also apparent that the results of the best 12-lead ECG computer programs proved to be almost as accurate as the best of seven cardiologists in classifying seven main disease categories, i.e., normal, left, right and biventricular hypertrophy, anterior, inferior and combined myocardial infarction. Evaluation of rhythm statements and conduction disturbances was not included in the study. The data collection is still being pursued in order to reach over 1,000 cases. In this way a common diagnostic database is being established for comparative testing of diagnostic computer programs. This should lead to consumer protection and improve the accuracy and reliability of computerized electrocardiography.

* For the CSE Working Party


 
  • REFERENCES

  • 1 Bruce RA, Yarnall SR. Reliability and normal variations of computer analysis of Frank electrocardiograms by the Smith-Hyde program. Am J Cardiol 1968; 29: 389-96.
  • 2 Yokoi M, Yasui S, Okajima M, Mizuno Y. Comparative studies on the diagnostic accuracy between the two diagnostic logic in electrocardiographic interpretation. Jpn Circ J 1969; 33: 51-8.
  • 3 Caceres CA, Hochberg HM. Performance of the computer and physician in the analysis of the electrocardiogram. Am Heart J 1970; 79: 439-43.
  • 4 Bailey JJ, Itscoitz SB, Hirshfield Jr JW, Grauer LE, Horton MB. A method for evaluating computer programs for electrocardiographic interpretation. I. Application to the experimental IBM program of 1971. Circulation 1974; 50: 73-9.
  • 5 Pipberger HV, Cornfield J. What ECG-computer program to choose for clinical application. The need for consumer protection. Circulation 1973; 47: 918-20.
  • 6 Meyer J, Heinrich KW, Merx W, Efferts S. Computeranalyse des Elektrocardiogramms mit verschiedenen Programmen. I. Formanalyse. Dtsch Med Wochenschr 1974; 23: 1213-23.
  • 7 Smets Ph, Willems JL, Talmon J, De Maertelaer V, Kornreich F. Methodology for the comparison of various diagnostic procedures. Biometrie-Praximetrie 1975; 15: 3-35.
  • 8 Dreifus LS, Caceres CA. How reliable is the computerized electrocardiographic interpretation?. JAMA 1975; 231: 1090.
  • 9 Pipberger HV. Comparative evaluation of ECG computer programs. In: Ripley KL, Ostrow HG. eds. Computers in Cardiology. Long Beach: IEEE Computer Society; 1976: 85-8.
  • 10 Willems JL, Piessens J, Ector J, Kesteloot H, De Geest H. Comparison of computer and visual ECG analysis results: correlation with independent cineangiographic data. In: Ostrow HG, Riply KL. eds. Computers in Cardiology. Long Beach: IEEE Computer Society; 1976: 89-94.
  • 11 Feruglio GA, Feraco E, Mirolo R. Experience of a user of an ECG interpretation system. In: van Bemmel JH, Willems JL. eds. Trends in Computer-Processed Electrocardiograms. Amsterdam: North-Holland, Publ Comp; 1977: 239-47.
  • 12 Rautaharju PM, Ariet M, Pryor TA. et al. The quest for optimal electrocardiography. Task Force III on computers in diagnostic electrocardiography. Am J Cardiol 1978; 41: 155-71.
  • 13 Bourdillon PJ, Kilpatrick D. Clinicians, the Mount Sinai program and the Veterans Administration program evaluated against clinico-pathological data derived independently of the electrocardiogram. Eur J Cardiol 1978; 08: 395-412.
  • 14 Rautaharju PM, Smets Ph. Evaluation of computer-ECG programs. The strange case of the golden standard. Comp Biomed Res 1979; 12: 39-46.
  • 15 Bonner R, Jackson LK, Hooper JK, Smith CH, Hsieh RC. A method for testing performance in electrocardiographic interpretation. In: Pryor AT, Bailey JJ. eds. Computerized Interpretation of the ECG, TV. New York: Engineering Foundation; 1979: 126-44.
  • 16 Brohet C, Richman HG. Clinical evaluation of automated processing of electrocardiograms by the Veterans Administration Program (AVA 3.4). Am J Cardiol 1979; 43: 1167-74.
  • 17 Bailey JJ, Harris EK. Evaluation of ECG interpretation: truth versus beauty. In: Tolan GD, Pryor TA. eds. Computerized Interpretation of the Electrocardiogram, V. New York: Engineering Foundation; 1980: 179-82.
  • 18 Macfarlane PW, Melville DI, Horton MR, Bailey JJ. Comparative evaluation of the IBM (12-Iead) and Royal Infirmary (orthogonal three-lead) ECG computer programs. Circulation 1981; 63: 354-9.
  • 19 Laks MM. “Gold standard” for ECG diagnosis - revisited 1981. In: Bonner RE, Pryor TA, Laks MM, Cole SS. eds. Computerized Interpretation of the ECG, VI. New York: Engineering Foundation; 1982: 267-75.
  • 20 Bailey JJ, Berson A, Jackson L, Milliken J, Stevens J, Tolan G, Wolf H. Evaluation methodologies for ECG diagnostic systems. In: Bonner RE, Pryor TA, Laks MM, Colle SS. eds. Computerized Interpretation of the ECG, VI. New York: Engineering Foundation; 1981: 53-62.
  • 21 Wolf HK, Milliken JA, Gregor RD, Burg-graf G, MacCahn J, Rautaharju PM, Cuddy TE. Accuracy of type “A” statements of human and computer ECG classifiers. In: Laks MM, Colle SS. eds. Computerized Interpretation of the Electrocardiogram, VII. New York: Engineering Foundation; 1982: 335-40.
  • 22 Bernard P, Chairman BR, Scholl JM, Val PG, Chabot M. Comparative diagnostic performance of the Telemed computer program. J Electrocardiol 1983; 16: 97-102.
  • 23 Graham IM, Reardon B, Mulcahy D, Mulcahy R. User evaluation of a commercially available computerized electrocardiogram interpretation system. In: Willems JL, van Bemmel JH, Zywietz C. eds. Computer ECG Analysis: Towards Standardization. Amsterdam: North-Holland, Publ Comp; 1986: 191-3.
  • 24 Willems JL. The CSE project: goals and achievements. In: Willems JL, van Bemmel JH, Zywietz C. ed. Computer ECG Analysis: Towards Standardisation. Amsterdam: North-Holland, Publ Comp; 1986: 11-21.
  • 25 Pipberger HV. ECG computer analysis: past, present and future. In: Willems JL, van Bemmel JH, Zywietz C. eds. Computer ECG Analysis: Towards Standardisation. Amsterdam: North-Holland; 1986: 3-10.
  • 26 Pipberger HV, McCaughan D, Littmann D. et al. Clinical application of a second generation electrocardiographic computer program. Am J Cardiol 1975; 35: 597-608.
  • 27 Harris LK, Blackburn HW, Prineas RJ, Wolf HK. Electrocardiographic diagnosis of myocardial infarction by human readers and two computer programs. In: Abstract Book I. European Congress on Cardiology; Paris: 1976: 725.
  • 28 Richman HG, Brohet CR. Evaluation of a statistical decision program for automated ECG processing. In: Hofrnan I, Hamby RE. eds. Vectorcardiography 3. Amsterdam: North-Holland Publ Comp; 1976: 199-207.
  • 29 Kornreich F, Block P, Bourgain R. et al. Multigroup diagnosis classification with a new “maximal” lead system. In: Abel H. ed. Advances in Cardiology and Electro-cardiology. Basel: S. Karger; 1976: 121-7.
  • 30 Khadr NE, Bray CL, Beton DC, Croxson RS, Hughes M, Jeffrey C, Richards B. Diagnosis of left ventricular hypertrophy and myocardial infarction by Bonner/IBM Program verified by ECG-independent evidence. In: Ripley KL, Ostrow HG. eds. Computers in Cardiology. Long Beach: IEEE Computer Society; 1979: 93-7.
  • 31 Brohet CR. Computer-Assisted Interpretation of Electro- and Vectorcardiograms. (Ph. D. Thesis). University of Louvain; 1980: 35-133.
  • 32 Willems JL, Lesaffre E, Pardaens J, De Schreye D. Multigroup logistic classification of the 12- and 3-lead ECG. In: Willems JL, van Bemmel JH, Zywietz C. eds. Computer ECG Analysis: Towards Standardisation. Amsterdam: North-Holland Publ Comp; 1986: 203-10.
  • 33 Willems JL, Lesaffre E, Pardaens J. Comparison of the classification ability of the electrocardiogram and vectorcardiogram. Am J Cardiol 1987; 59: 119-24.
  • 34 Willems JL, Arnaud P, van Bemmel JH, Degani R, Macfarlane PW, Zywietz C. Common standards for quantitative electrocardiography: goals and main results. Meth Inform Med 1990; 29: 263-71.
  • 35 van Bemmel JH, Willems JL. eds. Trends in Computer Processed Electrocardiograms. Amsterdam: North-Holland Publ Comp; 1977: 1-437.
  • 36 Willems JL. (CSE Project Leader). Common Standards for Quantitative Electrocardiography - 6th CSE Progress Report. Leuven: ACCO Publ; 1986: 1-269.
  • 37 Willems JL, Abreu-Lima C, Arnaud P, van Bemmel JH, Brohet CR. et al. Effect of combining electrocardiographic interpretation results on diagnostic accuracy. Eur Heart J 1988; 09: 1348-55.
  • 38 Austen WG, Edwards JE, Frye RL, Gensi-ni GG, Gott VL, Griffith LSC, McGoon DC, Murphy ML, Roe BB. Reporting system on patients evaluated for coronary artery disease. AHA Committee Report Circulation 1975; 51: 1-40.
  • 39 Macfarlane PW. A hybrid lead system for routine electrocardiography. In: Macfarlane PW. ed. Progress in Electrocardiography. London: Pittman Medical Publ; 1979: 1-5.
  • 40 Cornfield J, Dunn RA, Batchlor CD, Pipberger HV. Multigroup diagnosis of electrocardiograms. Comp Biomed Res 1973; 06: 97-120.
  • 41 Diamond GA, Hirsch M, Forrester JS. et al. Application of information theory to clinical diagnostic testing. The electrocardiographic stress test. Circulation 1981; 63: 915-21.
  • 42 Michaelis J, Willems JL. Performance evaluation of diagnostic ECG programs. In: Ripley KL. ed. Computers in Cardiology-Long Beach. California: IEEE Computer Society; 1988: 25-30.
  • 43 Bailey JJ, Campbell G, Horton MR, Shra-ger RI, Willems JL. Determination of statistically significant differences in the performance of ECG diagnostic algorithm: An improved method. J Electrocardiol 1988; 21: S188-S192.
  • 44 Willems JL, Campbell G, Bailey JJ. Progress on the CSE diagnostic study. Application of McNemar’s test revisited. J Electrocardiol 1989; 22: S136-S141.
  • 45 Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960; 20: 37-46.
  • 46 Crevasse L, Ariet M. A new scalar electrocardiographic computer program. Clinical evaluation. JAMA 1973; 226: 1089-93.
  • 47 Simonson E, Tuna N, Okamoto N. et al. Diagnostic accuracy of the electrocardiogram and vectorcardiogram. A cooperative study. Am J Cardiol 1966; 17: 829-78.
  • 48 Gjørup T. The Kappa coefficient and the prevalence of a diagnosis. Meth Inform Med 1988; 27: 184-6.
  • 49 Willems JL. Common Standards for Quantitative Electrocardiography - 9th CSE Progress Report. Leuven: Acco Publ; 1989: 1-318. (In press).
  • 50 Rautaharju PM, Blackburn HW, Wolf HK, Horacek M. Computers in clinical electrocardiology. Is vectorcardiography becoming obsolete?. In: Abel H. ed. Electrocardiology. Advances in Cardiology. Basel: Karger; 1976: 143-56.
  • 51 Dudeck J, Michaelis J. Problems in the diagnostic process in electrocardiography. In: Zywietz Chr, Schneider B. eds. Computer Application on ECG and VCG Analysis. Amsterdam: North-Holland Publ Comp; 1973: 283-95.