Download PDF
Methods Inf Med 2004; 43(03): 215-231
DOI: 10.1055/s-0038-1633862
Original Article
Schattauer GmbH

Future Directions in Evaluation Research: People, Organizational, and Social Issues

B. Kaplan
1   Yale University School of Medicine, New Haven, CT, USA
,
N. T. Shaw
2   THINK (The Health Informatics NetworK), Centre for Healthcare Innovation and Improvement, BC Research Institute for Children’s and Women’s Health, Vancouver, BC, Canada
› Author Affiliations
Further Information

Publication History

Publication Date:
05 February 2018 (online)

    Permissions and Reprints

Summary

Objective: To review evaluation literature concerning people, organizational, and social issues and provide recommendations for future research.

Method: Analyze this research and make recommendations.

Results and Conclusions: Evaluation research is key in identifying how people, organizational, and social issues – all crucial to system design, development, implementation, and use – interplay with informatics projects. Building on a long history of contributions and using a variety of methods, researchers continue developing evaluation theories and methods while producing significant interesting studies. We recommend that future research:

1) Address concerns of the many individuals involved in or affected by informatics applications.

2) Conduct studies in different type and size sites, and with different scopes of systems and different groups of users. Do multi-site or multi-system comparative studies.

3) Incorporate evaluation into all phases of a project.

4) Study failures, partial successes, and changes in project definition or outcome.

5) Employ evaluation approaches that take account of the shifting nature of health care and project environments, and do formative evaluations.

6) Incorporate people, social, organizational, cultural, and concomitant ethical issues into the mainstream of medical informatics.

7) Diversify research approaches and continue to develop new approaches.

8) Conduct investigations at different levels of analysis.

9) Integrate findings from different applications and contextual settings, different areas of health care, studies in other disciplines, and also work that is not published in traditional research outlets.

10) Develop and test theory to inform both further evaluation research and informatics practice.

Keywords

Evaluation - technology assessment - medical informatics - telemedicine - organizational culture - attitudes towards computers - implementation - barriers - people - organizational - social - issues - sociotechnical - ethical issues - qualitative methods - ethnographic methods - multi-method - human-computer interaction.
 

  • References

  • 1 Paré G, Sicotte C. Information technology sophistication in health care: An instrument validation study among Canadian hospitals. Int J Med Inform 2001; 63 (03) 205-23.
    CrossrefPubMedGoogle Scholar
  • 2 Kling R, Rosenbaum H, Hert C. Social informatics in information science: An introduction. J Am Soc Inform Sci 1998; 49 (12) 1047-52.
    CrossrefPubMedGoogle Scholar
  • 3 Berg M. Implementing information systems in health care organizations: Myths and challenges. Int J Med Inform 2001; 64 2-3 143-56.
    CrossrefPubMedGoogle Scholar
  • 4 Anderson JG, Jay SJ. (eds) Use and impact of computers in clinical medicine. New York: Springer; 1987
    Google Scholar
  • 5 Anderson JG, Aydin CE, Jay SJ. (eds) Evaluating health care information systems: Methods and applications. Thousand Oaks, CA: Sage Publications; 1994
    Google Scholar
  • 6 van Gennip EMS, Talmon JL. (eds) Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995
    Google Scholar
  • 7 Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York: Springer; 1997
    Google Scholar
  • 8 Rigby M. Evaluation: 16 powerful reasons why not to do it – and 6 over-riding imperatives. In Patel V, Rogers R, Haux R. (eds) Medinfo 2001. Amsterdam: IOS Press; 2001: 1198-201.
    Google Scholar
  • 9 Lorenzi NM, Riley RT. Organizational aspects of health informatics: Managing technological change. New York: Springer; 1995
    Google Scholar
  • 10 Lorenzi N, Riley RT. Managing change: An overview. JAMIA 2000; 7 (02) 116-24.
    PubMedGoogle Scholar
  • 11 Ash J. Managing change: Analysis of a hypothetical case. JAMIA 2000; 7 (02) 125-34.
    PubMedGoogle Scholar
  • 12 Kaplan B. Organizational evaluation of medical information systems. In Friedman CP, Wyatt JC. (eds) Evaluation methods in medical informatics. New York: Springer; 1997: 255-80.
    CrossrefGoogle Scholar
  • 13 Kaplan B. Addressing organizational issues into the evaluation of medical systems. JAMIA 1997; 4 (02) 94-101.
    PubMedGoogle Scholar
  • 14 Kuhn KA, Guise DA. From hospital information systems to health information systems – problems, challenges, perspectives. In Haux R, Kulikowski C. (eds) Yearbook of Medical Informatics. Stuttgart: Schattauer; 2001: 63-76.
    Google Scholar
  • 15 DoH, NHSE, IMG. Project review: Objective evaluation. Guidance for NHS managers on evaluating information systems projects; 1996.
    PubMedGoogle Scholar
  • 16 Ellis NT, Aarts J, Kaplan B, Leonard K. Alternatives to the controlled trial: New paradigms for evaluation. Proc AMIA Symp (CD ROM). 2000
    Google Scholar
  • 17 Goorman E, Berg M. Modelling nursing activities: Electronic patient records and their discontents. Nurs Inq 2000; 200 (07) 3-9.
    PubMedGoogle Scholar
  • 18 Greatbatch D, Murphy E, Dingwall R. Evaluating medical information systems: Ethnomethodological and interactionist approaches. Health Serv Manag Res 2001; 14 (03) 181-91.
    CrossrefPubMedGoogle Scholar
  • 19 Kaplan B, Aarts J, Hebert M, Klecun-Dabrowska E, Lewis D, Vimlarlund V. New approaches to evaluation: Alternatives to the randomized controlled trial – qualitative approaches to design and evaluation: Theory and practice. In Patel V, Rogers R, Haux R. (eds) Medinfo 2001. Amsterdam: IOS Press; 2001: 1531
    Google Scholar
  • 20 Kaplan B, Anderson JG, Jaffe C, Leonard K, Zitner D. New approaches to evaluation: Alternatives to the randomized controlled trial – qualitative models for evaluation. In Patel V, Rogers R, Haux R. (eds) Medinfo 2001. Amsterdam: IOS Press; 2001: 1456
    Google Scholar
  • 21 Kaplan B, Atkinson C, Ellis E, Jones M, Klecun-Dabrowska E, Teasdale S. Evaluation in the UK’s National Health Service. In Patel V, Rogers R, Haux R. (eds) Medinfo 2001. Amsterdam: IOS Press; 2001: 1529
    Google Scholar
  • 22 Kushniruk A. Evaluation in the design of health information systems: Application of approaches emerging from usability engineering. Comput Biol Med 2002; 32 (03) 141-49.
    CrossrefPubMedGoogle Scholar
  • 23 Protti D. A proposal to use a balanced scorecard to evaluate information for health: An information strategy for the modern NHS (1998-2005). Comput Biol Med 2002; 32 (03) 221-36.
    CrossrefPubMedGoogle Scholar
  • 24 Fischer PJ, Stratman WC, Lundsgaarde HP. User reactions to PROMIS: Issues related to acceptability of medical innovations. In O’Neill JT. (ed) Proc Symp Comput Appl Med Care. Silver Spring: IEEE Computer Society; 1980: 1722-30.
    Google Scholar
  • 24a Reprinted in Anderson JG, Jay SJ. (eds) Use and impact of computers in clinical medicine. New York: Springer; 1987: 284-301.
    Google Scholar
  • 25 Lundsgaarde HP, Fischer PJ, Steele DJ. Human problems in computerized medicine. Lawrence, KS: The University of Kansas; 1981
    Google Scholar
  • 26 Dlugacz YD, Siegel C, Fischer S. Receptivity towards uses of a new computer application in medicine. In Blum BL. (ed.) Proc Symp Comput Appl Med Care. Silver Spring: IEEE Computer Society; 1980: 384-91.
    Google Scholar
  • 27 Siegel C, Alexander MJ, Dlugacz YD, Fischer S. Evaluation of a computerized drug review system: Impact, attitudes and interactions. Comput Biomed Res 1984; 17: 419-35.
    CrossrefPubMedGoogle Scholar
  • 28 Anderson JG, Jay SL. Computers and clinical judgement: The role of physician networks. Soc Sci Med 1985; 20: 967-79.
    CrossrefPubMedGoogle Scholar
  • 29 Anderson JG, Jay SL, Schwerr HM, Anderson MM, Kassing D. Physician communication networks and the adoption and utilization of computer applications in medicine. In Anderson JG, Jay SJ. (eds) Use and impact of computers in clinical medicine. New York: Springer; 1987: 185-99.
    CrossrefGoogle Scholar
  • 30 Barrett JP, Barnum RA, Gordon BB, Pesut RN. Evaluation of a medical information system in a general community hospital. Report no. HSM 110-73-331, PB 233 784 (NTIS PB 248 340). Battelle Columbus Laboratories. 1975
    Google Scholar
  • 31 Kaplan BM. Computers in medicine, 1950-1980: The relationship between history and policy [Ph.D.]. University of Chicago; 1983
    Google Scholar
  • 32 Collen MF. A history of medical informatics in the United States: 1950-1990. Bethesda, MD: American Medical Informatics Association; 1995
    Google Scholar
  • 33 Farlee C, Goldstein B. Hospital organization and computer technology. New Brunswick, NJ: Health Care Systems Research; 1972
    Google Scholar
  • 34 Brown B, Harbot B, Kaplan B, Maxwell J. Guidelines for managing the implementation of automated medical systems. In Heffeman HG. (ed.) Proc Symp Comput Applic Med Care. Silver Spring: IEEE Computer Society Press; 1981: 791-96.
    Google Scholar
  • 35 Brown B, Harbot B, Kaplan B. Management issues in automating medical systems. J Clin Engr 1983; 8 (01) 23-30.
    PubMedGoogle Scholar
  • 36 Kaplan B. Barriers to medical computing history, diagnosis and therapy for the medical computing ‘lag’. In Ackerman MJ. (ed.) Proc Symp Comput Appl Med Care. Silver Spring: IEEE Computer Society Press; 1985: 400-4.
    Google Scholar
  • 37 Aydin CE. Occupational adaption to computerized medical information systems. J Health Soc Behav 1989; 30: 163-79.
    CrossrefPubMedGoogle Scholar
  • 38 Aydin CR, Rice RE. Bringing social worlds together: Computers as catalysts for new interactions in health care organizations. J Health Soc Behav 1992; 33: 168-85.
    CrossrefPubMedGoogle Scholar
  • 39 Aydin C. Computerized order entry in a large medical center: Evaluating interactions between departments. In Anderson JG, Aydin CE, Jay SJ. (eds) Evaluating health care information systems: Approaches and applications. Thousand Oaks, Cal: Sage; 1994: 260-75.
    Google Scholar
  • 40 Sigurdardottir H, Skov D, Bartholdy C, Wann-Larsen S. Evaluating the usefulness of monitoring change readiness in organisations which plan on implementing health informatics systems. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; 2001, September 6-7. Department of Health Policy and Management, Erasmus University; Rotterdam, The Netherlands: 2001: 52
    Google Scholar
  • 41 Hanmer JC. Diffusion of medical technologies: Comparison with ADP systems in the medical environment. In O’Neill JT. (ed) Proc Symp Comput Applic Med Care. Silver Spring: IEEE Computer Society Press; 1980: 1731-36.
    Google Scholar
  • 42 Kaplan B. Evaluating informatics applications – some alternative approaches: Theory, social interactionism, and call for methodological pluralism. Int J Med Inform 2001; 64: 39-56.
    CrossrefPubMedGoogle Scholar
  • 43 Kaplan B, Duchon D. Combining qualitative and quantitative approaches in information systems research: A case study. Manag Inform Sys Q 1988; 12 (04) 571-86.
    PubMedGoogle Scholar
  • 44 Kaplan B. A model comprehensive evaluation plan for complex information systems: Clinical imaging systems as an example. In Brown A, Remenyi D. (eds) Proc Second European Conference on Information Technology Investment Evaluation. Henley on Thames, Birmingham, England: Operational Research Society; 1995: 14-181.
    Google Scholar
  • 45 Kaplan B, Morelli R, Goethe J. Preliminary findings from an evaluation of the acceptability of an expert system in psychiatry. Extended Proc AMIA Symp (CD-ROM). 1997
    Google Scholar
  • 46 Degoulet Fieschi M. Critical dimensions in medical informatics. Int J Med Inform 1997; 44: 21-6.
    CrossrefPubMedGoogle Scholar
  • 47 Sicotte C, Denis JL, Lehoux P. The computer based patient record: A strategic issue in process innovation. J Med Systs 1998; 22 (06) 431-43.
    PubMedGoogle Scholar
  • 48 Safran C, Jones PC, Rind D, Booker B, Cytryn KN, Patel VL. Electronic communication and collaboration in a health care practice. Artif Intell Med 1998; 12 (02) 137-51.
    PubMedGoogle Scholar
  • 49 Ash JS, Lyman J, Carpenter J, Fournier L. A diffusion of innovations model of physician order entry. Proc AMIA Symp. 2001: 22-6.
    Google Scholar
  • 50 Mitev N, Kerkham S. Organization and implementation issues of patient data management systems in an intensive care unit. J End User Comput 2001; 13 (03) 20-9.
    CrossrefPubMedGoogle Scholar
  • 51 Wetter T. Lessons learned from bringing knowledge-based decision support into routine use. Artif Intell Med 2002; 24 (03) 195-203.
    CrossrefPubMedGoogle Scholar
  • 52 Lehoux P, Sicotte C, Denis JL, Berg M, Lacroix A. The theory of use behind telemedicine: How compatible with physicians’ clinical routines?. Soc Sci Med 2002; 54 (06) 889-904.
    CrossrefPubMedGoogle Scholar
  • 53 Hannan TJ, Rotich JK, Odero WW, Menya D, Esamai F, Einterz RM. et al. The Mosoriot medical record system; design and initial implementation of an outpatient electronic record system in rural Kenya. Int J Med Inform 2000; 60 (01) 21-8.
    CrossrefPubMedGoogle Scholar
  • 54 Wilson M, Howcroft D. The role of gender in user resistance and information systems failure. In Baskerville R, Stage J, DeGross JI. (eds) Organizational and social perspectives on information technology. Boston, Dordrecht, London: Kluwer Academic Publisher; 2000: 453-71.
    Google Scholar
  • 55 Novek J. IT, gender and professional practice: Or, why an automated drug distribution system was sent back to the manufacturer. Sci Technol Hum Val 2002; 27 (03) 379-403.
    CrossrefPubMedGoogle Scholar
  • 56 Kaplan B. Development and acceptance of medical information systems: An historical overview. J Health Hum Resour Admin 1988; 11 (01) 9-29.
    PubMedGoogle Scholar
  • 57 Massaro TA. Introducing physician order entry at a major academic medical center. 1: Impact on organizational culture and behavior. Acad Med 1993; 68 (01) 20-5.
    CrossrefPubMedGoogle Scholar
  • 58 Patel VL, Allen VG, Arocha JF, Shortliffe EH. Representing clinical guidelines in GLIF: Individual and collaborative expertise. JAMIA 1998; 5 (05) 467-83.
    PubMedGoogle Scholar
  • 59 Harrop VM. Virtual healthcare delivery: Defined, modeled and predictive barriers to implementation identified. Proc AMIA Symp. 2001: 244-8.
    Google Scholar
  • 60 May CR, Gask L, Atkinson T, Ellis NT, Mair F, Esmail A. Resisting and promoting new technologies in clinical practice: The case of ‘telepsychiatry’. Soc Sci Med 2001; 52 (12) 1889-901.
    CrossrefPubMedGoogle Scholar
  • 61 Kaplan B. The computer as Rorschach: Implications for management and user acceptance. In Dayhoff RE. (ed.) Proc Symp Comput Appl Med Care. Silver Spring: IEEE Computer Society Press; 1983: 664-67.
    Google Scholar
  • 62 van’t Reit A, Berg M, Hiddema F, Kees S. Meeting patients’ needs with patient information systems: Potential benefits from qualitative research methods. Int J Med Inform 2001; 64: 1-14.
    CrossrefPubMedGoogle Scholar
  • 63 Clarke K, O’Moore R, Smeets R, Talmon J, Brender J, McNair P. et al. Health systems evaluation of telemedicine: A staged approach. Telemed J 1996; 2 (04) 303-12.
    CrossrefPubMedGoogle Scholar
  • 64 Aarts J, Peel V, Wright G. Organizational issues in health informatics: A model approach. Int J Med Inf 1998; 52 1-3 235-42.
    CrossrefPubMedGoogle Scholar
  • 65 Folz-Murphy N, Partin M, Williams L, Harris CM, Lauer MS. Physician use of an ambulatory medical record system: Matching form and function. Proc AMIA Symp. 1998: 260-64.
    Google Scholar
  • 66 Nordyke RA, Kulikowski CA. An informatics based chronic disease practice – case study of a 35-year computer based longitudinal record. JAMIA 1998; 5 (01) 88-103.
    PubMedGoogle Scholar
  • 67 Dixon DR. The behavioral side of information technology. Int J Med Inform 1999; 56 1-3 117-23.
    CrossrefPubMedGoogle Scholar
  • 68 Manning B, Gadd CS. Introducing handheld computing into a residency program: Preliminary results from qualitative and quantitative inquiry. Proc. AMIA Symp. 2001: 428-32.
    Google Scholar
  • 69 Southon G. IT, change and evaluation: An overview of the role of evaluation in health services. Int J Med Inform 1999; 56 1-3 125-33.
    CrossrefPubMedGoogle Scholar
  • 70 Kaplan B. The influence of medical values and practices on medical computer applications. In Anderson JG, Jay SJ. (eds.) Use and impact of computers in clinical medicine. New York: Springer; 1987: 39-50.
    Google Scholar
  • 71 Heathfield HA, Wyatt J. Philosophies for the design and development of clinical decision-support systems. Methods Inf Med 1993; 32 (01) 1-8.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 72 Kaplan B. The computer prescription: Medical computing, public policy and views of history. Sci Technol Hum Val 1995; 20 (01) 5-38.
    CrossrefPubMedGoogle Scholar
  • 73 Forsythe DE. New bottles, old wine: Hidden cultural assumptions in a computerized explanation system for migraine sufferers. Med Anthropol Q 1996; 10 (04) 551-74.
    CrossrefPubMedGoogle Scholar
  • 74 Friedman CP. Information technology leadership in academic medical centers: A tale of four cultures. Acad Med 1999; 74 (07) 795-99.
    CrossrefPubMedGoogle Scholar
  • 75 Ash JS, Gorman PN, Lavelle M, Lyman J. Multiple perspectives on physician order entry. Proc AMIA Symp. 2000: 27-31.
    Google Scholar
  • 76 Whitley EA, Pouloudi A. Studying the translations of NHSnet. J End User Comput 2001; 13 (03) 31-40.
    PubMedGoogle Scholar
  • 77 Hannan TJ, Tierney WM, Rotich JK, Odero WW, Smith F, Mamlin JJ. et al. The Mosoriot medical record system (MMRS) phase I to phase II implementation: An outpatient computer-based medical record in rural Kenya. In Patel V, Rogers R, Haux R. editors Medinfo 2001. Amsterdam: IOS Press; 2001: 619-22.
    Google Scholar
  • 78 Tierney WM, Rotich JK, Smith FE, Bii J, Einterz RM, Hannan TJ. Crossing the ‘digital divide’: Implementing an electronic medical record system in a rural Kenyan health center to support clinical care and research. Proc AMIA Symp. 2002: 792-95.
    Google Scholar
  • 79 Brender J, McNair P. User requirements in a system development & evaluation context. Stud Health Technol Inform 2000; 77: 203-7.
    PubMedGoogle Scholar
  • 80 Bygholm A. End-user support: A necessary issue in the implementation and use of EPR systems. In Patel V, Rogers R, Haux R. (eds) Medinfo 2001. Amsterdam: IOS Press; 2001: 604-08.
    Google Scholar
  • 81 Brennan PF, Strombom I. Improving health care by understanding patient preferences: The role of computer technology. JAMIA 1998; 5 (03) 257-62.
    PubMedGoogle Scholar
  • 82 Graves WI, Nyce JM. Normative models and situated practice in medicine. Inform Decis Technol 1992; 18: 143-49.
    PubMedGoogle Scholar
  • 83 Nyce JM, Timpka T. Work, knowledge and argument in specialist consultations: Incorporating Incorporating tacit knowledge into system design and development. Med and Biol Eng and Comput 1993; 31 (01) HTA16-HTA19.
    PubMedGoogle Scholar
  • 84 Timpka T, Rauch E, Nyce JM. Towards productive knowledge-based systems in clinical organizations: A methods perspective. Artif Intell Med 1994; 6 (06) 501-19.
    CrossrefPubMedGoogle Scholar
  • 85 Novek J. Hospital pharmacy automation: Collective mobility or collective control?. Soc Sci Med 2000; 51 (04) 491-503.
    CrossrefPubMedGoogle Scholar
  • 86 Covvey HD. When is a system only a system? Healthcare Inform. Manag. Comm. Canada 2001; 15 (01) 46-7.
    PubMedGoogle Scholar
  • 87 Kaplan B, Brennan PF, Dowling AF, Friedman CP, Peel V. Towards an informatics research agenda: Key people and organizational issues. JAMIA 2001; 8 (03) 234-41.
    PubMedGoogle Scholar
  • 88 Leys M. Health care policy: Qualitative evidence and health technology assessment. Health Policy 2003; 65 (03) 217-26.
    CrossrefPubMedGoogle Scholar
  • 89 Lehoux P, Blume S. Technology assessment and the sociopolitics of health technologies. J Health Politics, Policy, and Law 2000; 25 (06) 1083-120.
    PubMedGoogle Scholar
  • 90 May C, Mort M, Williams T, Mair F, Gask L. Health technology assessment in its local contexts: Studies of telehealthcare. Soc Sci Med 2003; 57 (04) 697-710.
    CrossrefPubMedGoogle Scholar
  • 91 Farbey B, Land F, Targett D. The moving staircase – problems of appraisal and evaluation in a turbulent environment. Inform. Tech. People 1999; 12 (03) 238-52.
    PubMedGoogle Scholar
  • 92 Burkle T, Ammenwerth E, Prokosch HU, Dudeck J. Evaluation of clinical information systems. What can be evaluated and what cannot?. J Eval Clin Pract 2001; 7 (04) 373-85.
    CrossrefPubMedGoogle Scholar
  • 93 Land F. Evaluation in a socio-technical context. In Orlikowski WJ, Walsham G, Jones MR, DeGross JI. (eds.) Information technology and changes in organizational work. London: Chapman & Hall; 1996: 115-26.
    Google Scholar
  • 94 Friedman CF, Haug P. Report on conference track 5: Evaluation metrics and outcome. Int J Med Inform 2003; 69 2-3. 307-9.
    CrossrefPubMedGoogle Scholar
  • 95 Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of health information systems – problems and challenges. Int J Med Inform 2003; 71 2-3 125-35.
    CrossrefPubMedGoogle Scholar
  • 96 van Gennip EMS, Talmon JL. Introduction. In van Gennip EMS, Talmon JL. (eds) Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995: 1-5.
    Google Scholar
  • 97 Fallman H. A computer professor questions computerization: The projects will become institutions which no-one can evaluate. Lakartidningen 1997; 94 (04) 204-6.
    PubMedGoogle Scholar
  • 98 Green CJ, Moehr JR. Performance evaluation frameworks for vertically integrated health care systems: Shifting paradigms in Canada. Proc AMIA Symp. 2000: 315-19.
    Google Scholar
  • 99 Aarts J, Peel V. Using a descriptive model of change when implementing large scale clinical information systems to identify priorities for further research. Int J Med Inform 1999; 56 1-3 43-50.
    CrossrefPubMedGoogle Scholar
  • 100 Chismar WG, Wiley-Paton S. Test of the technology acceptance model for the Internet in pediatrics. Proc AMIA Symp. 2002; 155-59.
    PubMedGoogle Scholar
  • 101 Gagnon M-P, Godin G, Gagné C, Fortin J-P, Lamothe L, Reinharz D. et al. An adaptation of the theory of interpersonal behaviour to the study of telemedicine adoption by physicians. Int J Med Inform 2003; 71 2-3 103-15.
    CrossrefPubMedGoogle Scholar
  • 102 Grant A, Plante I, Leblanc F. The TEAM methodology for the evaluation of information systems in biomedicine. Comput Biol Med 2002; 32 (03) 195-207.
    CrossrefPubMedGoogle Scholar
  • 103 Kaplan B. Social interactionist framework for information systems studies: The 4C’s. In Larson TJ, Levine L, DeGross JI. (eds) IFIP WG8.2 & WG8.6 Joint Working Conference on Information Systems: Current issues and future changes: International Federation for Information Processing. 1998: 327-39. Available at www.bi.no/dep2/infomgt/wg82-86/proceedings/ accessed 4 Feb 2003
    Google Scholar
  • 104 Dwyer SJ, Templeton AW, Martin NL, Lee KR, Levine E, Batnitzky S. et al. The cost of managing digital diagnostic images. Radiology 1982; 144 (02) 313-18.
    CrossrefPubMedGoogle Scholar
  • 105 Crowe BL. Overview of some methodological problems in assessment of PACS. Int J Biomed Comput 1992; 30 3-4 181-86.
    CrossrefPubMedGoogle Scholar
  • 106 Kazanjian A, Green CJ. Beyond effectiveness: The evaluation of information systems using a comprehensive health technology assessment framwork. Comput Biol Med 2002; 32 (03) 165-77.
    CrossrefPubMedGoogle Scholar
  • 107 Lauer TW, Joshi K, Browdy T. Use of the equity implementation model to review clinical systems implementation efforts: A case report. JAMIA 2000; 7 (01) 91-102.
    PubMedGoogle Scholar
  • 108 Kokol P. A new microcomputer software system evaluation paradigm: The medical perspective. J Med Syst 1991; 15 (04) 269-75.
    CrossrefPubMedGoogle Scholar
  • 109 Kokol P. A tool for software and hardware evaluation. J Med Syst 1996; 20 (03) 167-72.
    CrossrefPubMedGoogle Scholar
  • 110 Shaw NT. ‘CHEATS’: A generic information communication technology (ICT) evaluation framework. Comput Biol Med 2002; 32 (03) 209-20.
    CrossrefPubMedGoogle Scholar
  • 111 Kaplan R, Norton D. The balanced score card – measures that drive performance. Harv Bus Rev 1992; 70 (01) 71-9.
    PubMedGoogle Scholar
  • 112 Gordon D, Chapman R, Kunov H, Dolan A, Carter M. Hospital management decision support: A balanced scorecard approach. In Cesnik B, McCray AT, Scherrer J-R. (eds) Medinfo ‘98. Amsterdam: IOS Press; 1998. p. Pt 1 453-56.
    Google Scholar
  • 113 Niss KU. The use of the balanced scorecard (BSC) in the model for investment and evaluation of medical information systems. Stud Health Technol Inform 1999; 68: 110-4.
    PubMedGoogle Scholar
  • 114 Heathfield H, Hudson P, Kay S, Roberts R, Williams J. Issues in the multidisciplinary assessment of healthcare information systems. Inform Tech People 1999; 12 (03) 253-75.
    PubMedGoogle Scholar
  • 115 Boloix G, Robillard P. A software system evaluation framework. Commun ACM 1995; 28 (12) 17-26.
    PubMedGoogle Scholar
  • 116 Cornford T, Doukidis GI, Forster D. Experience with a structure, process and outcome framework for evaluating an information system. Omega, Int J Manag Sci 1994; 255 (05) 491-504.
    PubMedGoogle Scholar
  • 117 Donabedian A. The quality of medical care: Methods for assessing and monitoring the quality of care for research and for quality assessment programs. Science 1978; 200: 856-64.
    CrossrefPubMedGoogle Scholar
  • 118 Donabedian A. Explorations in quality assessment and monitoring; Vol I: The definition of quality and approaches to its assessment. Ann Arbor: Health Adminisitration Press; 1980
    Google Scholar
  • 119 Hebert M. Telehealth success: Evaluation framework development. In Patel V, Rogers R, Haux R. (eds) Medinfo 2001. Amsterdam: IOS Press; 2001: 1145-49.
    Google Scholar
  • 120 DeLone W, McLean E. Information systems success: The quest for the dependent variable. Inform Syst Res 1992; 3 (01) 60-95.
    CrossrefPubMedGoogle Scholar
  • 121 Turunen P. A model for evaluation of health care information systems. In Remenyi D, Brown A. (eds) Procs 8th European Conference on Information Technology Evaluation September 17-18. 2001. Oxford, UK: Oriel College; 183-87.
    Google Scholar
  • 122 Turunen P, Salmela H. The internal stake holders’ attitudes toward the evaluation methods for medical information systems. In Pantzar E, Savolainen R, Tynjälä P. (eds) In search for a human-centred information society. Tampere: Tampere University Press; 2001: 253-65.
    Google Scholar
  • 123 Fineberg H, Bauan R, Sosman M. Computerized cranial tomography – effect on diagnostic and therapeutic plans. JAMIA 1997; 3: 244-7.
    PubMedGoogle Scholar
  • 124 van der Meijden MJ, Tange HH, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: A literature review. JAMIA 2003; 10 (03) 235-43.
    PubMedGoogle Scholar
  • 125 Krobock J. A taxonomy: Hospital information systems evaluation methodologies. J Med Systs 1984; 8 (05) 419-29.
    PubMedGoogle Scholar
  • 126 Grémy F, Degoulet P. Assessment of health information technology: Which questions for which systems? Proposal for a taxonomy. Med Inform (Lond) 1993; 18 (03) 185-93.
    PubMedGoogle Scholar
  • 127 Anderson JG, Aydin CE. Overview: Theoretical perspectives and methodologies for the evaluation of health care information systems. In Anderson JG, Aydin CE, Jay SJ. (eds) Evaluating health care information systems: Approaches and applications. Thousand Oaks, Cal: Sage; 1994: 5-29.
    Google Scholar
  • 128 Grant A, Buteau M, Richards Y, Delisle E, Laplante P, Niyonsenga T. et al. Informatics methodologies for evaluation research in the practice setting. Methods Inf Med 1998; 37 (02) 178-81.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 129 Talmon JL. Workshop WG15: Technology assessment and quality improvement. In Patel V, Rogers R, Haux R. (eds) Medinfo 2001. Amsterdam: IOS Press; 2001. 1540
    Google Scholar
  • 130 Jones MR. An interpretive method for the formative evaluation of an electronic patient record system. In Remenyi D, Brown A. (eds) Procs of the 8th European conference on IT evaluation 17-18 September. 2001. Oriel College; Oxford, UK: 349-56.
    Google Scholar
  • 131 May CR, Mort M, Williams T, Mair F, Shaw NT. Understanding the evaluation of telemedicine: The play of the social and the technical, and the shifting sands of reliable knowledge. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; September 6-7, 2001. Department of Health Policy and Management, Erasmus University; Rotterdam, The Netherlands: 2001: 43
    Google Scholar
  • 132 MacFarlane A, Harrison R, Wallace P. The benefits of a qualitative approach to telemedicine research. J Telemed Telecare 2002; 8 Suppl (Suppl. 02) 56-7.
    CrossrefPubMedGoogle Scholar
  • 133 May C, Williams T, Mair F MM, Shaw M, Gask L N. Factors influencing the evaluation of telehealth interventions: Preliminary results from a qualitative study of evaluation projects in the UK. J Telemed Telecare 2002; 8 Suppl (Suppl. 02) 65-7.
    CrossrefPubMedGoogle Scholar
  • 134 Williams T, May C, Mair F, Mort M, Shaw NT, Gask L. Normative models of health technology assessment and the social production of evidence about telehealth care. Health Policy 2003; 64 (01) 39-54.
    CrossrefPubMedGoogle Scholar
  • 135 Fulop N, Allen P, Clarke A, Black N. From health technology assessment to research on the organisation and delivery of health services: Addressing the balance. Health Policy 2003; 63 (02) 155-65.
    CrossrefPubMedGoogle Scholar
  • 136 Turunen P, Talmon J. Stakeholder groups in the evaluation of medical information systems. In Brown A, Remenyi D. (eds) Procs 7th European Conference on the Evaluation of Information Technology. September 28-29. 2000. Dublin, Ireland: 329-34.
    Google Scholar
  • 137 Chiasson MW, Davidson E. Pushing the contextual envelope: Developing and diffusing IS theory for health information systems research. Inform Organization 2004; in press.
    PubMedGoogle Scholar
  • 138 IT in Health Care: Sociotechnical Approaches. International Conference Proc. September Rotterdam, The Netherlands: Department of Health Policy and Management, Erasmus University; 2001: 6-7.
    Google Scholar
  • 139 Protti DJ, Haskell AR. Managing information in hospitals: 60% social, 40% technical. In: Proc IMIA Working Conference on Trends in Hospital Information Systems. 1992
    Google Scholar
  • 140 Lorenzi NM, Riley RT, Blyth AJC, Southon G, Dixon BJ. Antecedents of the people and organizational aspects of medical informatics: Review of the literature. JAMIA 1997; 4 (02) 79-93.
    PubMedGoogle Scholar
  • 141 Lundsgaarde H. Evaluating medical expert systems. Soc Sci Med 1987; 24 (10) 805-19.
    CrossrefPubMedGoogle Scholar
  • 142 Bosk C, Frader J. The impact of place of decision making on medical decisions. In O’Neill JT. (ed) Proc Symp Comput Applic Med Care. Silver Spring: IEEE Computer Society Press; 1980: 1326-29.
    Google Scholar
  • 143 Barley SR. Technology as an occasion for structuring: Evidence from observations of CT scanners and the social order of radiology departments. Admin Sci Q 1986; 31: 78-108.
    CrossrefPubMedGoogle Scholar
  • 144 Barley SR. The social construction of a machine: Ritual, superstition, magical thinking and other pragmatic responses to running a CT scanner. In Lock M, Gordon D. (eds) Biomedicine examined. Dordrecht: Kluwer Academic Publishers; 1988: 497-539.
    Google Scholar
  • 145 Barley SR. On technology, time, and social order: Technically induced change in the temporal organization of radiological work. In Dubinskas FA. (ed) Making time: Ethnographies of high-technology organizations. Philadelphia: Temple University Press; 1988: 123-69.
    Google Scholar
  • 146 Barley SR. The alignment of technology and structure through roles and networks. Admin Sci Q 1990; 35 (01) 61-103.
    CrossrefPubMedGoogle Scholar
  • 147 Barney S, Nyce JM, Ackerman M, Graves III WX, King J, Koh E. et al. Visualization in the neurosciences: Addressing problems in research, teaching and clinical practice. In: First Conference on Visualization in Biomedical Computing. Los Alamitos, Calif: IEEE Computer Society Press; 1990: 322-27.
    Google Scholar
  • 148 Nyce JM, Graves WI. The construction of knowledge in neurology: Implications for hypermedia system development. Artif Intell Med 1990; 29 (02) 315-22.
    PubMedGoogle Scholar
  • 149 Forsythe DE, Buchanan BG. Broadening our approach to evaluating medical information systems. In: Proc Symp Comput Appl Med Care 1991; 8-12.
    PubMedGoogle Scholar
  • 150 Saetnan AR. Rigid politics and technological flexibility; the anatomy of a failed hospital innovation. Sci Technol Hum Val 1991; 16 (04) 419-47.
    CrossrefPubMedGoogle Scholar
  • 151 Yoxen E. Seeing with sound: A study of the development of medical images. In Bijker WE, Hughes T, Pinch T. (eds.) The social construction of technological systems. Cambridge, Mass: MIT Press; 1987: 281-306.
    Google Scholar
  • 152 Forsythe DE. Using ethnography in the design of an explanation system. Expert Syst Applic 1995; 8 (04) 403-17.
    CrossrefPubMedGoogle Scholar
  • 153 Lau F, Hayward R. Building a virtual network in a community health research training program. JAMIA 2000; 7 (04) 361-77.
    PubMedGoogle Scholar
  • 154 Lee RG, Garvin T. Moving from information transfer to information exchange in health and health care. Soc Sci Med 2003; 56: 449-64.
    CrossrefPubMedGoogle Scholar
  • 155 Weaver RR. Evaluating the problem knowledge coupler: A case study. In Anderson JG, Aydin CE, Jay SJ. (eds) Evaluating health care information systems: Approaches and applications. Thousand Oaks, CA: Sage; 1994: 203-25.
    Google Scholar
  • 156 Ash J. Organizational factors that influence information technology diffusion in academic health sciences centers. JAMIA 1997; 4 (02) 102-11.
    PubMedGoogle Scholar
  • 157 Schubart JR, Einbinder JS. Evaluation of a data warehouse in an academic health sciences center. Int J Med Inform 2000; 60: 319-33.
    CrossrefPubMedGoogle Scholar
  • 158 Kaplan B. The medical computing ‘lag’: Perceptions of barriers to the application of computers to medicine. Int J Technol Assess Health Care 1987; 3 (01) 123-36.
    CrossrefPubMedGoogle Scholar
  • 159 Anderson JG, Aydin CE, Kaplan B. An analytical framework for measuring the effectiveness/impacts of computer-based patient record systems. In Nunamaker JF, Sprague RH. (eds) IV: Information Systems – collaboration systems and technology, organizational systems and technology. Los Alamitos, Cal: IEEE Computer Society Press; 1995: 767-76.
    Google Scholar
  • 160 Anderson JG. Evaluation in health informatics: Social network analysis. Comput Biol Med 2002; 32 (03) 179-93.
    CrossrefPubMedGoogle Scholar
  • 161 May C, Ellis NT. When protocols fail: technical evaluation, biomedical knowledge and the social production of ‘facts’ about a telemedicine clinic. Soc Sci Med 2001; 53 (08) 989-1002.
    CrossrefPubMedGoogle Scholar
  • 162 Sicotte C, Dennis JL, Lehoux P, Champagne F. The computer-based patient record: Challenges towards timeless and spaceless medical practice. J Med Systs 1998; 22 (04) 237-56.
    PubMedGoogle Scholar
  • 163 Kristensen M. Assessing a prerequisite for a constructivist approach – change readiness in organisations implementing health informatics systems. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc. 2001 6-7 September 2001. Department of Health Policy and Management, Erasmus University; Rotterdam, The Netherlands: 2001: 38
    Google Scholar
  • 164 Aarts J. On articulation and localization – some sociotechnical issues in design, implementation and evaluation of knowledge based systems. In Quaglini S, Barahona P, Andreassen S. (eds) Artificial Intelligence in Medicine: Proc. of the 8th Conference on Artificial Intelligence in Medicine in Europe AIME 2001. Berlin: Springer; 2001: 16-19.
    Google Scholar
  • 165 Berg M, Langenberg C, Berg I, Kwakkemaat J. Considerations for sociotechnical design: Experiences with an electronic patient record in a clinical context. Int J Med Inform 1998; 52 1-3 243-51.
    CrossrefPubMedGoogle Scholar
  • 166 Berg M, Goorman E. The contextual nature of medical information. Int J Med Inform 1999; 56 1-3 51-60.
    CrossrefPubMedGoogle Scholar
  • 167 Berg M. Patient care information systems and health care work: A sociotechnical approach. Int J Med Inform 1999; 55 (02) 87-101.
    CrossrefPubMedGoogle Scholar
  • 168 Reddy M, Pratt W, Dourish P, Shabot MM. Sociotechnical requirements analysis for clinical systems. Methods Inf Med 2003; 42 (04) 437-44.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 169 Stoop AP, Berg M. Integrating quantitative and qualitative methods in patient care information system evaluation: Guidance for the organizational decision maker. Methods Inf Med 2003; 42 (04) 458-62.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 170 Bygholm A. Activity theory as a framework for conducting end-use support. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; September 6-7, 2001. Department of Health Policy and Management, Erasmus University; Rotterdam, The Netherlands: 2001: 21
    Google Scholar
  • 171 Grémy F, Fessler JM, Bonnin M. Information systems evaluation and subjectivity. Int J Med Inform 1999; 56 1-3 13-23.
    CrossrefPubMedGoogle Scholar
  • 172 Davidson EJ. Analyzing genre of organizational communication in clinical information systems. Inform Tech People 2000; 33 (03) 196-209.
    PubMedGoogle Scholar
  • 173 Klecun-Dabrowska E, Cornford T. Evaluating telehealth: The search for an ethical perspective. Camb Q Healthc Ethics 2001; 10 (02) 161-69.
    CrossrefPubMedGoogle Scholar
  • 174 Zerubavel E. Patterns of time in hospital life. Chicago and London: University of Chicago Press; 1979
    Google Scholar
  • 175 Ash J, Berg M. Report of conference track 4: Socio-technical issues of HIS. Int J Med Inform 2003; 69 2-3 305-6.
    CrossrefPubMedGoogle Scholar
  • 176 Mathiassen L. Collaborative research practice. In Orlikowski WJ, Walsham G, Jones MR, DeGross JI. (eds) Information technology and changes in organizational work. London: Chapman & Hall; 1996: 127-48.
    Google Scholar
  • 177 Sawyer S, Rosenbaum H. Social informatics in the information sciences: Current activities and emerging directions. Inform Sci 2002; 3: 2
    PubMedGoogle Scholar
  • 178 Reddy M, Pratt W, Dourish P, Shabot MM. Asking questions: Information needs in a surgical intensive care unit. Proc AMIA Symp. 2002: 647-51.
    Google Scholar
  • 179 Methods Inf Med. 2003; 42: 4
    PubMedGoogle Scholar
  • 180 Kaplan B, Lau F, Aarts J, Forsythe DE. Information systems qualitative research in health care. In Lee AS, Liebenau J, DeGross JI. (eds) Information systems and qualitative research. London: Chapman & Hall; 1997: 569-70.
    Google Scholar
  • 181 Aarts J, Goorman E, Heathfield H, Kaplan B. Successful development, implementation and evaluation of information systems: Does healthcare serve as a model for networked organizations?. In Baskerville R, Stage J, DeGross JI. editors Organizational and social perspective on information technology. Boston, Dordrecht, London: Kluwer Academic Publishers; 2000: 517-20.
    Google Scholar
  • 182 Kaplan B, Kvasny L, Sawyer S, Trauth EM. New words and old books: Challenging conventional discourses about domain and theory in information systems research. In Myers MD, Whitley EA, Wynn E, De Gross JI. (eds) Global and organizational discourse about information technology. London: Kluwer; 2002: 539-45.
    Google Scholar
  • 183 Bloomfield B, Coombs R, Cooper D, Rea D. Machines and manoeuvres: Responsibility accounting and the construction of hospital information systems. Accounting Manag Inform Technol 1992; 2: 197-219.
    PubMedGoogle Scholar
  • 184 Bloomfield BP, McLean C. Madness and organization: Informed management and empowerment. In Orlikowski WJ, Walsham G, Jones MR, DeGross JI. (eds) Information technology and changes in organizational work. London: Chapman & Hall; 1996: 371-93.
    Google Scholar
  • 185 Inform Tech People. 1999; 12: 3
    PubMedGoogle Scholar
  • 186 Silva L, Backhouse J. Becoming part of the furniture: The institutionalization of infromation systems. In Lee AS, Leibenau J, DeGross JI. (eds) Information systems and qualitative research. London: Chapman & Hall; 1997: 389-414.
    Google Scholar
  • 187 J End User Comput. 2001; 13: 3-4.
    PubMedGoogle Scholar
  • 188 Walsham G. IT and changing professional identity: Micro-studies and macro-theory. J Am Soc Inform Sci 1998; 49: 1081-89.
    CrossrefPubMedGoogle Scholar
  • 189 Avgerou C. Information systems and global diversity. Oxford: Oxford University Press; 2002
    Google Scholar
  • 190 Hersh WR, Patterson PK, Kraemer DF. Telehealth: The need for evaluation redux. JAMA 2002; 9 (01) 89-91.
    PubMedGoogle Scholar
  • 191 Nykanen P, Chowdhury S, Wigertz O. Evaluation of decision support systems in medicine. Comput Methods Programs Biomed 1991; 34: 229-38.
    CrossrefPubMedGoogle Scholar
  • 192 Allen D. Telemental health services today. Telemed Today 1994; 2 (02) 1-24.
    PubMedGoogle Scholar
  • 193 Burghgraeve P, De Maeseneer J. Improved methods for assessing information technology in primary health care and an example from telemedicine. J Telemed Telecare 1995; 1: 157-64.
    CrossrefPubMedGoogle Scholar
  • 194 Bashur R. On the definition and evaluation of telemedicine. Telemed J 1995; 1 (01) 19-30.
    CrossrefPubMedGoogle Scholar
  • 195 Filiberti D, Wallace J, Koteeswaran R, Neft D. A telemedicine transaction model. Telemed J 1995; 1 (03) 237-47.
    CrossrefPubMedGoogle Scholar
  • 196 Grigsby J. You got an attitude problem or what?. Telemed Today. 1995: 32-4.
    Google Scholar
  • 197 Grigsby J, Schlenker R, Kaehny M, Shaughnessy P. Analytic framework for evaluation of telemedicine. Telemed J 1995; 1 (01) 31-9.
    CrossrefPubMedGoogle Scholar
  • 198 Grigsby J. Sentenced to life without parole doing outcomes research. Telemed Today. 1996: 40-1.
    Google Scholar
  • 199 DeChant H, Tohme W, Mun S, Hayes W. Health systems evaluation of telemedicine: A staged approach. Telemed J 1996; 2 (04) 303-12.
    CrossrefPubMedGoogle Scholar
  • 200 McIntosh E, Cairns J. A framework for the economic evaluation of telemedicine. J Telemed Telecare 1997; 3: 132-9.
    CrossrefPubMedGoogle Scholar
  • 201 Yellowlees P. Practical evaluation of telemedicine systems in the real world. J. Telemed. Telecare 1998; 4 S1 56-7.
    CrossrefPubMedGoogle Scholar
  • 202 Perednia DM. Telemedicine system evaluation, transaction models and multicentered research. J Am Health Inform Manag Assn 1996; 67 (01) 60-3.
    PubMedGoogle Scholar
  • 203 Rigby M. Health informatics as a tool to improve quality in non-acute care – new opportunities and a matching need for a new evaluation paradigm. Int J Med Inform 1999; 56 1-3 141-50.
    CrossrefPubMedGoogle Scholar
  • 204 Tohme WG, Hayes WS, Dai H, Komo D. Evaluation of a telemedicine platform for three medical applications. Comp Assisted Rad. 1996: 505-11.
    Google Scholar
  • 205 Wyatt JC. Commentary: Telemedicine trials – clinical pull or technology push?. BMJ 1996; 313: 380-1.
    PubMedGoogle Scholar
  • 206 May C, Gask L, Ellis NT, Atkinson TA, Mair F, Smith C. et al. Telepsychiatry evaluation in the North West of England: Preliminary results. Telemed Telecare 2000; 6 S1 20-2.
    PubMedGoogle Scholar
  • 207 Qavi T, Corley L, Kay S. Nursing staff requirements for telemedicine in the neonatal intensive care unit. J End User Comput 2001; 13 (03) 5-13.
    PubMedGoogle Scholar
  • 208 Reiser SJ. Medicine and the reign of technology. Cambridge: Cambridge University Press; 1978
    Google Scholar
  • 209 Kaplan B, Brennan PF. Consumer informatics supporting patients as co-producers of quality. JAMIA 2001; 8 (04) 309-16.
    PubMedGoogle Scholar
  • 210 Mort M, May CR, Williams T. Remote doctors and absent patients: Acting at a distance in telemedicine?. Sci Technol Hum Val 2003; 28 (02) 274-95.
    CrossrefPubMedGoogle Scholar
  • 211 Turner J. Telemedicine: Generating the virtual office visit. In Eder LB. (ed) Managing healthcare information systems with web-enabled technologies. Hershey and London: Idea Group Publishing; 2000: 59-68.
    Google Scholar
  • 212 Balas EA, Jaffrey F, Kuperman GJ, Boren SA, Brown GD, Pinciroli F. et al. Electronic communication with patients. Evaluation of distance medicine technology. JAMA 1997; 278 (02) 152-9.
    CrossrefPubMedGoogle Scholar
  • 213 Finkelstein J, Cabrera MR, Hripcsak G. Internetbased home asthma telemonitoring: Can patients handle the technology?. Chest 2000; 117 (01) 148-55.
    CrossrefPubMedGoogle Scholar
  • 214 Sandelowski M. Visible humans, vanishing bodies, and virtual nursing: Complications of life, presence, place, and identity. Advances in Nursing Science 2002; 24 (03) 58-70.
    CrossrefPubMedGoogle Scholar
  • 215 Allaert FA, Weinberg D, Dusserre P, Yvon PJ, Dusserre L, Cotran P. Evaluation of a telepathology system between Boston (USA) and Dijon (France): Glass slides versus telediagnostic tvmonitor. Proc Annu Symp Comput Appl Med Care. 1995: 596-600.
    Google Scholar
  • 216 Allaert FA, Weinberg D, Dusserre P, Yvon PJ, Dusserre L, Retaillau B. et al. Evaluation of an international telepathology system between Boston (USA) and Dijon: Glass slides versus telediagnostic television monitor. J Telemed Telecare 1996; 2 Suppl (Suppl. 01) 27-30.
    CrossrefPubMedGoogle Scholar
  • 217 Travers DA, Downs SM. Comparing the user acceptance of a computer system in two pediatric offices: A qualitative study. Proc AMIA Symp. 2000: 853-7.
    Google Scholar
  • 218 Kaplan B. Objectification and negotiation in interrupting clinical images: Implications for computerbased patient records. Artif Intell Med 1995; 7: 439-54.
    CrossrefPubMedGoogle Scholar
  • 219 Kaplan B, Lundsgaarde HP. Toward an evaluation of a clinical imaging system: Identifying benefits. Methods Inf Med 1996; 35: 221-9.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 220 Billinghurst M, Hirokazu K. Collaborative augmented reality. Commun ACM 2002; 45 (07) 64-70.
    PubMedGoogle Scholar
  • 221 Moehr JR. Evaluation: Salvation or nemesis of medical informatics?. Comput Biol Med 2002; 32 (03) 113-25.
    CrossrefPubMedGoogle Scholar
  • 222 May CR, Mort M, Mair F, Ellis NT, Gask L. Evaluation of new technologies in health care systems: What’s the context?. Health Inform J 2000; 6 (02) 67-70.
    CrossrefPubMedGoogle Scholar
  • 223 Harrison R, MacFarlane A, Wallace P. Implementation of telemedicine: The problem of evaluation. J Telemed Telecare 2002; 8 Suppl (Suppl. 02) 39-40.
    CrossrefPubMedGoogle Scholar
  • 224 Pinch TJ, Bijker WE. The social construction of facts and artifacts: Or how the sociology of science and the sociology of technology might benefit each other. In Bijker W, Hughes T, Pinch T. (eds) The social construction of technological systems. Cambridge, Mass: MIT Press; 1987: 17-50.
    Google Scholar
  • 225 Nøhr C. The evaluation of expert diagnostic systems – how to assess outcomes and quality parameters?. Artif Intell Med 1994; 6 (02) 123-35.
    CrossrefPubMedGoogle Scholar
  • 226 Grémy F, Bonnin M. Evaluation of automatic health information systems. In van Gennip EMS, Talmon JL. editors Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995: 9-20.
    Google Scholar
  • 227 Jørgensen T. Measuring effects. In van Gennip EMS, Talmon JL. (eds) Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995: 99-109.
    Google Scholar
  • 228 Miller RA. Medical diagnostic decision support systems – past, present and future – a threaded bibliography and brief commentary. JAMIA 1994; 1: 8-27.
    PubMedGoogle Scholar
  • 229 Heathfield HA, Buchan IE. Current evaluations of information technology in health care are often inadequate. BMJ. 1996; 313 7063 1008
    PubMedGoogle Scholar
  • 230 Gosling AS, Westbrook JI, Coiera EW. Variation in the use of online clinical evidence: A qualitative analysis. Int J Med Inform 2003; (01) 1-16.
    PubMedGoogle Scholar
  • 231 Friedman RB, Gustafson DH. Computers in clinical medicine, a critical review. Comput Biomed Res 1977; 10 (03) 199-204.
    CrossrefPubMedGoogle Scholar
  • 232 Bankowitz RA, Lave JR, McNeil MA. A method for assessing the impact of a computer-based decision support system on health care outcomes. Methods Inf Med 1992; 31 (01) 3-10.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 233 Goldberg HS, Morales A, Gottlieb L, Meador L, Safran C. Reinventing patient-centered computing for the twenty-first century. In Patel V, Rogers R, Haux R. (eds.) Medinfo 2001. Amsterdam: IOS Press; 2001: 1455-58.
    Google Scholar
  • 234 Kreps GL. Evaluating new health information technologies: Expanding the frontiers of health care delivery and health promotion. Stud Health Technol Inform 2002; 80: 205-12.
    PubMedGoogle Scholar
  • 235 Pugh GE, Tan JK. Computerized databases for emergency care: What impact on patient care?. Methods Inf Med 1994; 33 (05) 507-13.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 236 Banta H. Embracing or rejecting innovations: Clinical diffusion of health care technology. In Anderson JG, Jay SJ. (eds) Use and impact of computers in clinical care. New York: Springer; 1987: 132-60.
    Google Scholar
  • 237 Miller RA. Why the standard view is standard: People, not machines, understand patients’ problems. J Med Philos 1990; 15 (06) 581-91.
    CrossrefPubMedGoogle Scholar
  • 238 Harris KD, Donaldson JF, Campbell JD. Introducing computer-based telemedicine in three rural Missouri counties. J End User Comput 2001; 13 (04) 26
    PubMedGoogle Scholar
  • 239 Penrod LE, Gadd CS. Attitudes of academicbased and community-based physicians regarding EMR use during outpatient encounters. Proc AMIA Symp. 2001: 529-32.
    Google Scholar
  • 240 Ash JS, Gorman PN, Hersh WR. Physician order entry in U.S. Hospitals. Proc AMIA Symp. 1998: 235-9.
    Google Scholar
  • 241 Ash JS, Gorman PN, Hersh WR, Poulson SP. Perceptions of house officers who use physician order entry. Proc AMIA Symp. 1999: 471-5.
    Google Scholar
  • 242 Ash J, Gorman P, Lavelle M, Lyman J, Fournier L. Investigating physician order entry in the field: Lessons learned in a multi-centered study. In Patel V, Rogers R, Haux R. (eds) Medinfo 2001. Amsterdam: IOS Press; 2001: 1107-11.
    Google Scholar
  • 243 Ash JS, Gorman PN, Lavelle M, Stavri PZ, Lyman J, Fournier L. et al. Perceptions of physician order entry: Results of a cross-site qualitative study. Methods Inf Med 2003; 42 (04) 313-23.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 244 Ash JS, Gorman PN, Lavelle M, Payne TH, Massaro TA, Frantz GL. et al. A cross-site qualitative study of physician order entry. JAMIA 2003; 10 (02) 188-200.
    PubMedGoogle Scholar
  • 245 Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerations for a sucessful CPOE implementation. JAMIA 2003; 10 (03) 229-34.
    PubMedGoogle Scholar
  • 246 Coombs CR, Doherty NF, Loan-Clark J. The importance of user ownership and positive user attitudes in the successful adoption of community information systems. J End User Comput 2001; 13 (04) 5
    PubMedGoogle Scholar
  • 247 Lau F, Doze S, Vincent D, Wilson D, Noseworthy T, Hayward R. Patterns of improvisation for evidencebased practice in clinical settings. Inform Tech People 1999; 12 (03) 287-303.
    PubMedGoogle Scholar
  • 248 Isaacs S. The power distance between users of information technology and experts and satisfaction with the information system. In Patel V, Rogers R, Haux R. (eds) Medinfo 2001. Amsterdam: IOS Press; 2001: 1155-7.
    Google Scholar
  • 249 Talmon J, Enning J, Castãneda G, Eurlings F, Hoyer D, Nykänen P. et al. The VATAM guidelines. Int J Med Inform 1999; 56 1-3 107-15.
    CrossrefPubMedGoogle Scholar
  • 250 Suchman LA. Plans and situated actions: The problem of human-machine interaction. Cambridge: Cambridge University Press; 1987
    Google Scholar
  • 251 Suchman L. Representations of work. Commun ACM 1995; 38 (09) 33-4.
    PubMedGoogle Scholar
  • 252 Butler KA, Esposito C, Hebron R. Connecting the design of software to the design of work. Commun ACM 1999; 42 (01) 38-46.
    PubMedGoogle Scholar
  • 253 Fafchamps D, Young CY, Tang PC. Modelling work practices: Input to the design of a physician’s workstation. In Clayton PD. (ed.) Proc Symp Comput Appl Med Care. New York: McGraw Hill; 1991: 788-92.
    Google Scholar
  • 254 Holtzblatt K, Bryer HR. Apprenticing with the customer. Commun ACM 1995; 28 (01) 45-52.
    PubMedGoogle Scholar
  • 255 Artif Intell Med. 1995: 7
    Google Scholar
  • 256 Greenbaum J, Kyng M. Design at work: Cooperative design of computer systems. Hillsdale, NJ: Lawrence Erlbaum Associates; 1991
    Google Scholar
  • 257 Winograd T, Flores F. Understanding computers and cognition: A new foundation for design. Norwood, NJ: Ablex; 1986
    Google Scholar
  • 258 Timpka T, Hedblom P, Holmgren J. Action design: Using an object oriented environment for group process development of medical software. In Timmers T. (ed.) Software engineering in medical informatics. Amsterdam: Elsevier; 1991: 151-65.
    Google Scholar
  • 259 Sjöberg C, Timpka T. Participatory design of information systems in health care. J Am Health Inform Manag Assn 1998; 5 (02) 177-83.
    PubMedGoogle Scholar
  • 260 Musen MA. Architectures for architects. Methods Inf Med 1993; 32: 12-3.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 261 Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: Cognitive approaches to evaluation of information systems and user interfaces. Proc AMIA Symp. 1997; 218-22.
    PubMedGoogle Scholar
  • 262 Kushniruk AW, Patel VL. Cognitive evaluation of decision making processes and assessment of information technology in medicine. Int J Med Inform 1998; 51 2-3 83-90.
    CrossrefPubMedGoogle Scholar
  • 263 Patel V, Kaufman D. Medical informatics and the science of cognition. JAMIA 1998; 5 (06) 493-502.
    PubMedGoogle Scholar
  • 264 Beuscart-Zéphir MC, Anceaux AF, Crinquette C, Renarda JM. Integrating users’ activity modeling in the design and assessment of hospital electronic patient records. Int J Med Inform 2001; 64 2-3 157-71.
    CrossrefPubMedGoogle Scholar
  • 265 Kaplan B, Drickamer M, Marattoli RA. Deriving design recommendations through discount usability engineering: Ethnographic observation and thinking-aloud protocol in usability testing for computer-based teaching cases. Proc AMIA Symp. 2003: 346-50.
    Google Scholar
  • 266 Gardner RM, Lundsgaarde HP. Evaluation of user acceptance of a clinical expert system. JAMIA 1994; 1 (06) 428-38.
    PubMedGoogle Scholar
  • 267 Saranummi N. Supporting system development with technology assessment. In van Gennip EMS, Talmon JL. (eds) Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995: 35-44.
    Google Scholar
  • 268 Faber M. Design and introduction of an electronic patient record: How to involve users?. Methods Inf Med 2003; 42 (04) 371-5.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 269 Smith ME. Success rates for different types of organizational change. Performance Improvement 2002; 41 (01) 26-33.
    PubMedGoogle Scholar
  • 270 Lau F, Hebert M. Experiences from health informatics information system implementation projects reported in Canada between 1991 and 1997. J End User Comput 2001; 13 (04) 17
    CrossrefPubMedGoogle Scholar
  • 271 Dowling AF. Do hospital staff interfere with computer system implementation?. Health Care Manag Rev 1980; 5: 23-32.
    CrossrefPubMedGoogle Scholar
  • 272 Amatayakul M. The state of the computer-based patient record. JAHIMA 1998; 69: 34
    PubMedGoogle Scholar
  • 273 Dambro MR, Weiss BD, McClure C, Vuturo AF. An unsuccessful experience with computerized medical records in an academic medical centre. J Med Ed 1988; 63: 617-23.
    PubMedGoogle Scholar
  • 274 Flowers S. Software failure, management failure: Amazing stories and cautionary tales. Wiley. 1986
    Google Scholar
  • 275 Chessare JB, Torok KE. Implementation of COSTAR in an academic group practice of general pediatrics. MD Comput 1993; 10 (01) 23-27.
    PubMedGoogle Scholar
  • 276 Tonnesen AS, LeMaistre A, Tucker D. Electronic medical record implementation barriers encountered during implementation. Proc AMIA Symp. 1999: 625-66.
    Google Scholar
  • 277 Jeffcott M. Technology alone will never work: Understanding how organizational issues contribute to user neglect and information systems failure in healthcare. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; September 6-7, 2001. Department of Health Policy and Management, Erasmus University; Rotterdam, The Netherlands: 2001: 35
    Google Scholar
  • 278 Zuiderent T. Changing care and theory: A theoretical joyride as the safest journey. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; September 6-7, 2001. Department of Health Policy and Management, Erasmus University; Rotterdam, The Netherlands: 2001: 62
    Google Scholar
  • 279 Southon F, Sauer C, Dampney C. Information technology in complex health services: Organizational impediments to successful technology transfer and diffusion. JAMIA 1997; 4: 112-24.
    PubMedGoogle Scholar
  • 280 Balka E. Getting the big picture: The macro-politics of information system development (and failure) in a Canadian hospital. Methods Inf Med 2003; 42 (04) 324-30.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 281 Benyon-Davies P. The London ambulance service’s computerised dispatch system: A case study in information system failure: University of Glamorgan Pontyprid. 1993
    Google Scholar
  • 282 Benyon-Davies P. Information systems ‘failure’: The case of the London ambulance service’s computer aided despatch project. Eur J Inform Syst 1995; 4: 171-84.
    CrossrefPubMedGoogle Scholar
  • 283 Friedman CP, Wyatt JC. Publication bias in medical informatics. JAMIA 2001; 8 (02) 189-91.
    PubMedGoogle Scholar
  • 284 Keen P. Information systems and organizational change. Commun ACM 1981; 24 (01) 24-33.
    CrossrefPubMedGoogle Scholar
  • 285 Kling R, Iacono S. The mobilization of support for computerization: The role of computerization movements. Soc Problems 1988; 35 (03) 226-43.
    CrossrefPubMedGoogle Scholar
  • 286 Stead WW. Matching the level of evaluation to a project’s stage of development. JAMIA 1996; 3 (01) 92-94.
    PubMedGoogle Scholar
  • 287 Forsythe DE. The construction of work in artificial intelligence. In Hess DJ. (eds) Studying those who study us: An anthropologist in the world of artificial intelligence. Stanford, Calif: Stanford University Press; 2001: 16-34.
    Google Scholar
  • 288 Kaplan B. Evaluating informatics applications – clinical decision support systems literature review. Int J Med Inform 2001; 64 (01) 15-37.
    CrossrefPubMedGoogle Scholar
  • 289 Markus M. Power, politics, and MIS implementation. Commun ACM 1983; 26: 430-44.
    CrossrefPubMedGoogle Scholar
  • 290 Barsade SG, Brief AP, Spartaro S. The affective revolution of organizational behavior: The emergence of a paradigm. In Greenberg J. (ed) OB: The state of the science. Second ed. Hillsdale, NJ: L. Erlbaum Associates; 2002
    Google Scholar
  • 291 Forsythe DE, Buchanan B, Osheroff J, Miller R. Expanding the concept of medical information: An obsevational study of physicians’ information needs. Comput Biomed Res 1992; 25: 181-200.
    CrossrefPubMedGoogle Scholar
  • 292 Kaplan B, Farzanfar R, Friedman RH. Personal relationships with an intelligent interactive telephone health behavior advisor system: A multimethod study using surveys and ethnographic interviews. Int J Med Inform 2003; 71 (01) 33-41.
    CrossrefPubMedGoogle Scholar
  • 293 Tierney WM, Overhage JM, McDonald CJ. A plea for controlled trials in medical informatics. JAMIA 1994; 1 (04) 353-5.
    PubMedGoogle Scholar
  • 294 Wyatt JC, Wyatt SM. When and how to evaluate health information systems?. Int J Med Inform 2003; 69 2-3 251-9.
    CrossrefPubMedGoogle Scholar
  • 295 Mair F, Whitten P. Systematic review of studies of patient satisfaction with telemedicine. BMJ 2000; 320 7248 1517-20.
    CrossrefPubMedGoogle Scholar
  • 296 Banta D. The development of health technology assessment. Health Policy 2003; 63 (02) 121-32.
    CrossrefPubMedGoogle Scholar
  • 297 van der Loo RP. Overview of published assessment and evaluation studies. In van Gennip EMS, Talmon JL. (eds.) Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995: 261-82.
    Google Scholar
  • 298 Heathfield HA, Peel V, Hudson P, Kay S, Mackay L, Marley T. et al. Evaluating large scale health information systems: From practice towards theory. Proc AMIA Symp. 1997: 116-20.
    Google Scholar
  • 299 Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: Barriers and challenges. BMJ 1998; 316 7149 1959-61.
    CrossrefPubMedGoogle Scholar
  • 300 Paré G. Implementing clinical information systems: A multiple-case study within a US hospital. Health Ser Manag Res 2002; 15 (02) 71-92.
    PubMedGoogle Scholar
  • 301 Kaplan B, Maxwell JA. Qualitative research methods for evaluating computer information systems. In Anderson JG, Aydin CE, Jay SJ. (eds) Evaluating health care information systems; approaches and applications. Thousand Oaks, CA: Sage; 1994: 45-68.
    Google Scholar
  • 302 Sawyer S, Eschenfelder KR. Social informatics: Perspectives, examples and trends. In Cronon B. (eds) Annual review of information technology. Medford NJ: 2002: 427-65.
    Google Scholar