Evaluation – The Cinderella Science of ICT in Health

Summary

Objectives

Information systems are expensive to create, and consume resources in their operation. They are justified in order to achieve clearly defined objectives in health service quality and efficiency. To ensure that these objectives are being met, and in order to build up an evidence-base to support well focused policies, systems, and practice, evaluation would seem to be fully justified and indeed essential. However, there are major factors jeopardising good evaluation practice, ranging from reluctance to commit resources or expose decisions to scrutiny, through to lack of appreciation of the full range of dimensions which should be studied. This paper seeks to create fuller appreciation of the issues and their importance.

Methods

Experiential and literature reviews form the basis of an exposition of principles, methods, and key current activities.

Results

A summary has been produced of the key challenges to health ICT evaluation, and its adverse comparison with other sectors. An explanation of the viewpoints and levels of health information evaluation is followed by a short history of principal milestones, current scientific methodology activity, and key collaborative activities.

Conclusion

The need for a stronger commitment to health ICT evaluation is demonstrated if the application of health informatics systems is to receive due recognition as scientifically and empirically justified. Commitment of resources and scientific endeavour are needed, and thus the recent Declaration of Innsbruck, arising out of an event sponsored by the European Science Foundation, is timely and fully justified.

Keywords

Evaluation - information systems

 

• References

• 1 Rigby M. Evaluation: 16 Powerful Reasons Why Not to Do It – And 6 Over-Riding Imperatives. In: Patel V, Rogers R, Haux R. editors. Medinfo 2001: Proceedings of the 10^th World Congress on Medical Informatics. Amsterdam: IOS Press; 2001: 11198-202.
• 2 Guist SHF, Rigby MJ. The Rise and Fall of the Innovator. In: Lun KC, Degoulet P, Piemme TE, Rienhoff O. Medinfo 92 - Proceedings of the Seventh World Congress on Medical Informatics, Geneva Palexpo, Switzerland, 6-10 September 1992. Amsterdarm: Elsevier Science Publications BV, North-Holland; 1992
• 3 http://iig.umit.at/efmi/ (Bad Health Informatics page) (accessed 21.11.2005)
• 4 Ash J, Coiera E, Berg M. Some Unintended Consequences of Information Technology in Health Care: The Nature of Patient Care Information System-related Errors. J AmMed Inform Assoc 2004; (11) 104-12.
• 5 Institute of Medicine. Dick R, Steen E. editors The Computer Based Patient Record An – Essential Technology for Healthcare. Washington DC: National Academy Press; 1991
• 6 Department of Health. Information for Health: An Information Strategy for the Modern NHS. Department of Health; London: 1998
• 7 Department of Health. Delivering 21st Century IT Support for the NHS. London: Department of Health; 2004
• 8 National Health Information Management Advisory Council. Health Online. A Health Information Action Plan for Australia. Canberra: Department of Health and Aged Care; y1999.;
• 9 Rigby M. Health Informatics as a Tool to Improve Quality in Non-acute Care New Opportunities and a Matching Need for a – New Evaluation Paradigm. Int J Med Inform 1999; (56) 141-50.
• 10 Malmquist G, Hansson U, Qvarnström Å, Carlsson G. Health Practice by Remote Expert: a Case Study from Sweden. In: Rigby M, Roberts R, Thick M. editors. Taking Health Telematics into the 21^st. Century. Abingdon: Radcliffe Medical Press; 2000
• 11 Anderson JG, Aydin CE, Jay SJ. editors. Evaluating Health Care Information Systems: Methods and Applications. Thousand Oaks, CA: Sage; 1994
• 12 van Gennip EMSJ, Talmon JL. editors. Assessment and Evaluation of Information Technologies in Medicine. Amsterdam: IOS Press; 1995
• 13 Hasselbring W, Peterson R, Smits M, Spanjers R. Strategic Information Management for a Dutch University Hospital. In: Hasman A, Blobel B, Dudeck D, Engelbrecht R, Gell G, Prokosch H-U. editors. Medical Infobahn for Europe: Proceedings of MIE2000 and GMDS2000. Amsterdam: IOS Press; 2000: 969-73.
• 14 Aas IH. Telemedical work and co-operation. J Telemed Telecare 2001; 07 (04) 212-8.
• 15 Aas IH. A qualitative study of the organizational consequences of telemedicine. J Telemed Telecare 2001; 07 (01) 18-26.
• 16 Aas IH. Changes in the job situation due to telemedicine. J Telemed Telecare 2002; 08: 41-7.
• 17 Aas IH. Organizing for remote consultations in health care – the production process. Behaviour and Information Technology 2003; 22: 91-100.
• 18 Aas IH. Organizational cooperation in teleradiology. J Telemed Telecare 2005; 11: 45-50.
• 19 www.esf.org/generic/1651/EW0212Report.pdf (accessed 22.11.2005).
• 20 Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J. et al. Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inform 2004; 73 (06) 479-91.
• 21 http://iig.umit.at/efmi/ (accessed 21.11.05).
• 22 www.imia.org (follow Working Groups link) (accessed 21.11.2005).
• 23 Grémy F, Bonnin M. Evaluation of Automatic Health Information Systems – What and How?. In: van Gennip EMSJ, Talmon JL. editors. Assessment and Evaluation of Information Technologies in Medicine. Amsterdam: IOS Press; 1995
• 24 Grémy F. Hardware, software, peopleware, subjectivity. A philosophical promenade. Methods Inf Med 2005; 44 (03) 352-8.
• 25 Institute of Medicine. Crossing the Quality Chasm: a new health system for the 21^st . Century. Washington DC: National Academy Press; 2001
• 26 Friedman CP, Wyatt J C. Evaluation Methods in Medical Informatics. New York: Springer; 1997
• 27 Beuscart-Zephir MC, Brender J, Beuscart R, Menager-Depriester I. Cognitive evaluation: how to assess the usability of information technology in healthcare. Comp Methods Programs Biomed 1997; 54 (1-2): 19-28.
• 28 Beuscart-Zephir MC, Anceaux F, Renard JM. Integrating users’ activity analysis in the design and assessment of medical software applications: the example of anesthesia. Stud Health Technol Inform 2000; 77: 234-8.
• 29 Beuscart-Zephir MC, Menu H, Evrard F, Guerlinger S, Watbled L, Anceaux F. Multidimensional evaluation of a Clinical Information System for anaesthesiology: quality management, usability, and performances. Stud Health Technol Inform 2003; 95: 649-54.
• 30 Beuscart-Zephir MC, Pelayo S, Degoulet P, Anceaux F, Guerlinger S, Meaux JJ. A usability study of CPOE’s medication administration functions: impact on physician-nurse cooperation. Medinfo 2004; 11 (Pt 2): 1018-22.
• 31 Rigby M. Essential Prerequisites to the Safe and Effective Widespread Roll-out of E-Working in Healthcare. Int J Med Inform 2006; 75: 138-47.
• 32 Roger-France F, Noothoven van JGoor, Staehr-Johansen K. Case-Based Telematic Systems Towards Equity in Health Care (Studies in Health Technology and Informatics Vol. 14). Amsterdam: IOS Press; 1994
• 33 Lorenzi NM, Riley RT, Ball MJ, Douglas JV. Transforming Health Care Through Information – Case Studies. New York: Springer-Verlag; 1995
• 34 Drazen EL, Metzger JB, Ritter JL, Schneider MK. Patient Care Infomration Systems – Successful design and Implementation. New York: Springer-Verlag; 1995
• 35 http://evaldb.umit.at/ (accessed 22.11.2005).
• 36 Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002. Methods Inf Med 2005; 44 (01) 44-56.
• 37 Brender J, Ammenwerth E, Nykänen P, Talmon J. Factors Influencing Success and Failure of Health Informatics Systems – A Pilot Delphi Study. Methods Inf Med 2006; 45 (01) 125-36.
• 38 Brender J. Handbook of Evaluation Methods for Health Informatics. Academic Press; 2006
• 39 Talmon JL, Ammenwerth E. The declaration of Innsbruck: some reflections. Stud Health Technol Inform 2004; 110: 68-74.
• 40 Ammenwerth E, Shaw NT. Bad health informatics can kill—is evaluation the answer?. Methods Inf Med 2005; 44 (01) 1-3.