Download PDF
Methods Inf Med 1990; 29(04): 308-316
DOI: 10.1055/s-0038-1634794
ECG Interpretation Systems
Schattauer GmbH

Evaluation of ECG Interpretation Results Obtained by Computer and Cardiologists[*]

J. L. Willems
1   CSE Coordinating Center, Univ. of Leuven, Belgium
,
C. Abreu-Lima
2   Department of Cardiology, University of Porto, Portugal
,
P. Arnaud
3   INSEPvM, Unité 121, Hôp. Cardiovasculaire, Lyon, France
,
C. R. Brohet
4   Department of Cardiology, University of Louvain, Brussels, Belgium
,
B. Denis
5   Service de Cardiologie, Univ. of Grenoble, France
,
J. Gehring
6   Clinic for Cardiovascular Diseases, Höhenried, Federal Republic of Germany
,
I. Graham
7   Adelaide Hospital, Dublin, Ireland
,
G. van Herpen
8   University Hospital, Leiden, The Netherlands
,
H. Machado
9   University Hospital Santa Maria, Lisbon, Portugal
,
J. Michaelis
10   Institut für Medizinische Statistik und Dokumentation, Mainz, Federal Republic of Germany
,
S. D. Moulopoulos
11   Department of Therapeutics, University of Athens, Athens, Greece
› Author AffiliationsThe research presented in this paper has been supported by the European Commission within the frame of its 2nd, 3rd and 4th Medical and Public Health Research Programmes of DG XII, Section Biomedical Engineering, and by various funding agencies in ten Member States of the European Community. The authors are grateful to Ms. I. Tassens for secretarial assistance and to Mr. I. Schoolmeesters for computer programming help.
Further Information

Publication History

Publication Date:
06 February 2018 (online)

    Permissions and Reprints

Abstract

In an international project investigators from 25 institutes are trying to establish a common reference library and evaluation methods for testing the diagnostic performance of various ECG computer programs and of cardiologists, based on ECG-independent clinical information. A first set of 500 validated ECGs was collected and analyzed by fifteen different computer programs and nine cardiologists, seven of who analysed the ECG and five the VCG. A coding scheme was used to map individual diagnostic statements onto a common set. Combined program and referee results were obtained by weighted averaging. Preliminary results indicate that the classification accuracy of several programs can still be improved. However, it was also apparent that the results of the best 12-lead ECG computer programs proved to be almost as accurate as the best of seven cardiologists in classifying seven main disease categories, i.e., normal, left, right and biventricular hypertrophy, anterior, inferior and combined myocardial infarction. Evaluation of rhythm statements and conduction disturbances was not included in the study. The data collection is still being pursued in order to reach over 1,000 cases. In this way a common diagnostic database is being established for comparative testing of diagnostic computer programs. This should lead to consumer protection and improve the accuracy and reliability of computerized electrocardiography.

Key-Words

ECG Classification - Computerized Electrocardiography - Diagnostic Accuracy - Performance Testing

* For the CSE Working Party


 

  • REFERENCES

  • 1 Bruce RA, Yarnall SR. Reliability and normal variations of computer analysis of Frank electrocardiograms by the Smith-Hyde program. Am J Cardiol 1968; 29: 389-96.
    PubMedGoogle Scholar
  • 2 Yokoi M, Yasui S, Okajima M, Mizuno Y. Comparative studies on the diagnostic accuracy between the two diagnostic logic in electrocardiographic interpretation. Jpn Circ J 1969; 33: 51-8.
    CrossrefPubMedGoogle Scholar
  • 3 Caceres CA, Hochberg HM. Performance of the computer and physician in the analysis of the electrocardiogram. Am Heart J 1970; 79: 439-43.
    CrossrefPubMedGoogle Scholar
  • 4 Bailey JJ, Itscoitz SB, Hirshfield Jr JW, Grauer LE, Horton MB. A method for evaluating computer programs for electrocardiographic interpretation. I. Application to the experimental IBM program of 1971. Circulation 1974; 50: 73-9.
    CrossrefPubMedGoogle Scholar
  • 5 Pipberger HV, Cornfield J. What ECG-computer program to choose for clinical application. The need for consumer protection. Circulation 1973; 47: 918-20.
    CrossrefPubMedGoogle Scholar
  • 6 Meyer J, Heinrich KW, Merx W, Efferts S. Computeranalyse des Elektrocardiogramms mit verschiedenen Programmen. I. Formanalyse. Dtsch Med Wochenschr 1974; 23: 1213-23.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 7 Smets Ph, Willems JL, Talmon J, De Maertelaer V, Kornreich F. Methodology for the comparison of various diagnostic procedures. Biometrie-Praximetrie 1975; 15: 3-35.
    PubMedGoogle Scholar
  • 8 Dreifus LS, Caceres CA. How reliable is the computerized electrocardiographic interpretation?. JAMA 1975; 231: 1090.
    CrossrefPubMedGoogle Scholar
  • 9 Pipberger HV. Comparative evaluation of ECG computer programs. In: Ripley KL, Ostrow HG. eds. Computers in Cardiology. Long Beach: IEEE Computer Society; 1976: 85-8.
    Google Scholar
  • 10 Willems JL, Piessens J, Ector J, Kesteloot H, De Geest H. Comparison of computer and visual ECG analysis results: correlation with independent cineangiographic data. In: Ostrow HG, Riply KL. eds. Computers in Cardiology. Long Beach: IEEE Computer Society; 1976: 89-94.
    Google Scholar
  • 11 Feruglio GA, Feraco E, Mirolo R. Experience of a user of an ECG interpretation system. In: van Bemmel JH, Willems JL. eds. Trends in Computer-Processed Electrocardiograms. Amsterdam: North-Holland, Publ Comp; 1977: 239-47.
    Google Scholar
  • 12 Rautaharju PM, Ariet M, Pryor TA. et al. The quest for optimal electrocardiography. Task Force III on computers in diagnostic electrocardiography. Am J Cardiol 1978; 41: 155-71.
    PubMedGoogle Scholar
  • 13 Bourdillon PJ, Kilpatrick D. Clinicians, the Mount Sinai program and the Veterans Administration program evaluated against clinico-pathological data derived independently of the electrocardiogram. Eur J Cardiol 1978; 08: 395-412.
    PubMedGoogle Scholar
  • 14 Rautaharju PM, Smets Ph. Evaluation of computer-ECG programs. The strange case of the golden standard. Comp Biomed Res 1979; 12: 39-46.
    CrossrefPubMedGoogle Scholar
  • 15 Bonner R, Jackson LK, Hooper JK, Smith CH, Hsieh RC. A method for testing performance in electrocardiographic interpretation. In: Pryor AT, Bailey JJ. eds. Computerized Interpretation of the ECG, TV. New York: Engineering Foundation; 1979: 126-44.
    Google Scholar
  • 16 Brohet C, Richman HG. Clinical evaluation of automated processing of electrocardiograms by the Veterans Administration Program (AVA 3.4). Am J Cardiol 1979; 43: 1167-74.
    CrossrefPubMedGoogle Scholar
  • 17 Bailey JJ, Harris EK. Evaluation of ECG interpretation: truth versus beauty. In: Tolan GD, Pryor TA. eds. Computerized Interpretation of the Electrocardiogram, V. New York: Engineering Foundation; 1980: 179-82.
    Google Scholar
  • 18 Macfarlane PW, Melville DI, Horton MR, Bailey JJ. Comparative evaluation of the IBM (12-Iead) and Royal Infirmary (orthogonal three-lead) ECG computer programs. Circulation 1981; 63: 354-9.
    CrossrefPubMedGoogle Scholar
  • 19 Laks MM. “Gold standard” for ECG diagnosis - revisited 1981. In: Bonner RE, Pryor TA, Laks MM, Cole SS. eds. Computerized Interpretation of the ECG, VI. New York: Engineering Foundation; 1982: 267-75.
    Google Scholar
  • 20 Bailey JJ, Berson A, Jackson L, Milliken J, Stevens J, Tolan G, Wolf H. Evaluation methodologies for ECG diagnostic systems. In: Bonner RE, Pryor TA, Laks MM, Colle SS. eds. Computerized Interpretation of the ECG, VI. New York: Engineering Foundation; 1981: 53-62.
    Google Scholar
  • 21 Wolf HK, Milliken JA, Gregor RD, Burg-graf G, MacCahn J, Rautaharju PM, Cuddy TE. Accuracy of type “A” statements of human and computer ECG classifiers. In: Laks MM, Colle SS. eds. Computerized Interpretation of the Electrocardiogram, VII. New York: Engineering Foundation; 1982: 335-40.
    Google Scholar
  • 22 Bernard P, Chairman BR, Scholl JM, Val PG, Chabot M. Comparative diagnostic performance of the Telemed computer program. J Electrocardiol 1983; 16: 97-102.
    CrossrefPubMedGoogle Scholar
  • 23 Graham IM, Reardon B, Mulcahy D, Mulcahy R. User evaluation of a commercially available computerized electrocardiogram interpretation system. In: Willems JL, van Bemmel JH, Zywietz C. eds. Computer ECG Analysis: Towards Standardization. Amsterdam: North-Holland, Publ Comp; 1986: 191-3.
    Google Scholar
  • 24 Willems JL. The CSE project: goals and achievements. In: Willems JL, van Bemmel JH, Zywietz C. ed. Computer ECG Analysis: Towards Standardisation. Amsterdam: North-Holland, Publ Comp; 1986: 11-21.
    Google Scholar
  • 25 Pipberger HV. ECG computer analysis: past, present and future. In: Willems JL, van Bemmel JH, Zywietz C. eds. Computer ECG Analysis: Towards Standardisation. Amsterdam: North-Holland; 1986: 3-10.
    Google Scholar
  • 26 Pipberger HV, McCaughan D, Littmann D. et al. Clinical application of a second generation electrocardiographic computer program. Am J Cardiol 1975; 35: 597-608.
    CrossrefPubMedGoogle Scholar
  • 27 Harris LK, Blackburn HW, Prineas RJ, Wolf HK. Electrocardiographic diagnosis of myocardial infarction by human readers and two computer programs. In: Abstract Book I. European Congress on Cardiology; Paris: 1976: 725.
    Google Scholar
  • 28 Richman HG, Brohet CR. Evaluation of a statistical decision program for automated ECG processing. In: Hofrnan I, Hamby RE. eds. Vectorcardiography 3. Amsterdam: North-Holland Publ Comp; 1976: 199-207.
    Google Scholar
  • 29 Kornreich F, Block P, Bourgain R. et al. Multigroup diagnosis classification with a new “maximal” lead system. In: Abel H. ed. Advances in Cardiology and Electro-cardiology. Basel: S. Karger; 1976: 121-7.
    Google Scholar
  • 30 Khadr NE, Bray CL, Beton DC, Croxson RS, Hughes M, Jeffrey C, Richards B. Diagnosis of left ventricular hypertrophy and myocardial infarction by Bonner/IBM Program verified by ECG-independent evidence. In: Ripley KL, Ostrow HG. eds. Computers in Cardiology. Long Beach: IEEE Computer Society; 1979: 93-7.
    Google Scholar
  • 31 Brohet CR. Computer-Assisted Interpretation of Electro- and Vectorcardiograms. (Ph. D. Thesis). University of Louvain; 1980: 35-133.
    Google Scholar
  • 32 Willems JL, Lesaffre E, Pardaens J, De Schreye D. Multigroup logistic classification of the 12- and 3-lead ECG. In: Willems JL, van Bemmel JH, Zywietz C. eds. Computer ECG Analysis: Towards Standardisation. Amsterdam: North-Holland Publ Comp; 1986: 203-10.
    Google Scholar
  • 33 Willems JL, Lesaffre E, Pardaens J. Comparison of the classification ability of the electrocardiogram and vectorcardiogram. Am J Cardiol 1987; 59: 119-24.
    CrossrefPubMedGoogle Scholar
  • 34 Willems JL, Arnaud P, van Bemmel JH, Degani R, Macfarlane PW, Zywietz C. Common standards for quantitative electrocardiography: goals and main results. Meth Inform Med 1990; 29: 263-71.
    PubMedGoogle Scholar
  • 35 van Bemmel JH, Willems JL. eds. Trends in Computer Processed Electrocardiograms. Amsterdam: North-Holland Publ Comp; 1977: 1-437.
    Google Scholar
  • 36 Willems JL. (CSE Project Leader). Common Standards for Quantitative Electrocardiography - 6th CSE Progress Report. Leuven: ACCO Publ; 1986: 1-269.
    Google Scholar
  • 37 Willems JL, Abreu-Lima C, Arnaud P, van Bemmel JH, Brohet CR. et al. Effect of combining electrocardiographic interpretation results on diagnostic accuracy. Eur Heart J 1988; 09: 1348-55.
    CrossrefPubMedGoogle Scholar
  • 38 Austen WG, Edwards JE, Frye RL, Gensi-ni GG, Gott VL, Griffith LSC, McGoon DC, Murphy ML, Roe BB. Reporting system on patients evaluated for coronary artery disease. AHA Committee Report Circulation 1975; 51: 1-40.
    PubMedGoogle Scholar
  • 39 Macfarlane PW. A hybrid lead system for routine electrocardiography. In: Macfarlane PW. ed. Progress in Electrocardiography. London: Pittman Medical Publ; 1979: 1-5.
    Google Scholar
  • 40 Cornfield J, Dunn RA, Batchlor CD, Pipberger HV. Multigroup diagnosis of electrocardiograms. Comp Biomed Res 1973; 06: 97-120.
    CrossrefPubMedGoogle Scholar
  • 41 Diamond GA, Hirsch M, Forrester JS. et al. Application of information theory to clinical diagnostic testing. The electrocardiographic stress test. Circulation 1981; 63: 915-21.
    CrossrefPubMedGoogle Scholar
  • 42 Michaelis J, Willems JL. Performance evaluation of diagnostic ECG programs. In: Ripley KL. ed. Computers in Cardiology-Long Beach. California: IEEE Computer Society; 1988: 25-30.
    Google Scholar
  • 43 Bailey JJ, Campbell G, Horton MR, Shra-ger RI, Willems JL. Determination of statistically significant differences in the performance of ECG diagnostic algorithm: An improved method. J Electrocardiol 1988; 21: S188-S192.
    CrossrefPubMedGoogle Scholar
  • 44 Willems JL, Campbell G, Bailey JJ. Progress on the CSE diagnostic study. Application of McNemar’s test revisited. J Electrocardiol 1989; 22: S136-S141.
    PubMedGoogle Scholar
  • 45 Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960; 20: 37-46.
    CrossrefPubMedGoogle Scholar
  • 46 Crevasse L, Ariet M. A new scalar electrocardiographic computer program. Clinical evaluation. JAMA 1973; 226: 1089-93.
    CrossrefPubMedGoogle Scholar
  • 47 Simonson E, Tuna N, Okamoto N. et al. Diagnostic accuracy of the electrocardiogram and vectorcardiogram. A cooperative study. Am J Cardiol 1966; 17: 829-78.
    CrossrefPubMedGoogle Scholar
  • 48 Gjørup T. The Kappa coefficient and the prevalence of a diagnosis. Meth Inform Med 1988; 27: 184-6.
    PubMedGoogle Scholar
  • 49 Willems JL. Common Standards for Quantitative Electrocardiography - 9th CSE Progress Report. Leuven: Acco Publ; 1989: 1-318. (In press).
    Google Scholar
  • 50 Rautaharju PM, Blackburn HW, Wolf HK, Horacek M. Computers in clinical electrocardiology. Is vectorcardiography becoming obsolete?. In: Abel H. ed. Electrocardiology. Advances in Cardiology. Basel: Karger; 1976: 143-56.
    Google Scholar
  • 51 Dudeck J, Michaelis J. Problems in the diagnostic process in electrocardiography. In: Zywietz Chr, Schneider B. eds. Computer Application on ECG and VCG Analysis. Amsterdam: North-Holland Publ Comp; 1973: 283-95.
    Google Scholar