People, Organizational, and Social Issues: Evaluation as an exemplar

B. Kaplan; N.T. Shaw

doi:10.1055/s-0038-1638129

Yearbook of Medical Informatics, Table of Contents

Yearb Med Inform 2002; 11(01): 91-102
DOI: 10.1055/s-0038-1638129

Review

Georg Thieme Verlag KG

People, Organizational, and Social Issues: Evaluation as an exemplar

Authors

B. Kaplan

¹Yale University School of Medicine New Haven, CT, USA Kaplan Associates, Hamden, CT, USA
N.T. Shaw

²Lancashire Postgraduate School of Medicine & Health, UCLAN Preston, Lancashire, UK

References
1 Kaplan B, Brennan PF, Dowling AF, Friedman CP, Peel V. Towards an informatics research agenda: key people and organizational issues. J Am Med Inform Assoc 2001; May-June 8 (03) 234-41.
2 Dowling Jr AF. Do hospital staff interfere with computer system implementation?. Health Care Management Review 1980; 5: 23-32.
2a Reprinted in Anderson JG, Jay SJ. editors Use and impact of computers in clinical medicine. New York: Springer; 1987: 302-17.
3 Kuhn KA, Guise DA. From hospital information systems to health information systems-problems, challenges, perspectives. In. Haux R, Kulikowski C. editors Yearbook of Medical Informatics. Stuttgart: Schattauer; 2001: 63-76.
4 Brender J, Nøhr C, McNair P. Research needs and priorities in health informatics. Int J Med Inform 2000; Sep 58 (59) 257-89.
5 Ellis NT. Telemedicine and information for health: the impediments to implementation. In HC’99 Information At The Heart Of Clinical Practice. March. Harrogate: 1999: 22-24.
6 Kaplan B. The medical computing ‘lag’: perceptions of barriers to the application of computers to medicine. Int J Technol Assess Health Care 1987; 3 (01) 123-36.
7 Anderson JG, Aydin CE. Overview: theoretical perspectives and methodologies for the evaluation of health care information systems. In Anderson JG, Aydin CE, Jay SJ. Editors Evaluating health care information systems: Approaches and applications. Sage: Thousand Oaks, Cal; 1994: 5-29.
8 Kaplan B. Organizational evaluation of medical information systems. In Friedman CP, Wyatt JC. Evaluation methods in medical informatics. NY: Springer; 1997: 255-80.
9 Lauer TW, Joshi K, Browdy T. Use of the equity implementation model to review clinical systems implementation efforts: a case report. J Am Med Inform Assoc 2000; Jan/Feb 7 (01) 91-102.
10 Gadd CS, Penrod LE. Assessing physician attitudes regarding use of an outpatient EMR: a longitudinal, multipractice study. Proc AMIA Symp. 2001: 194-8.
11 Kaplan B. The influence of medical values and practices on medical computer applications. In Proc, MEDCOMP ‘82: The First IEEE Computer Society International Conference on Medical Computer Science/Computational Medicine. Silver Spring, Md: IEEE Computer Society Press; 1982: 83-8.
11a Reprinted. in Anderson JG, Jay SJ. Eds Use and impact of computers in clinical medicine. New York: Springer; 1987: 39-50.
12 Kaplan B. Culture counts: how institutional values affect computer use. MD Computing 2000; Jan/Feb 17 (01) 23-6.
13 Friedman CP. Information technology leadership in academic medical centers: a tale of four cultures. Acad Med 1999; 74 (07) 795-9.
14 Kaplan B. User acceptance of medical computer applications: a diffusion approach. In Blum BI. editor Proc, Symp Computer Applications in Medical Care. Silver Spring, Md: IEEE Computer Society Press; 1982. 6 398-402.
15 Kaplan B. The computer as Rorschach: implications for management and user acceptance. In Dayhoff RE. editor Proc, Symp Computer Applications in Medical Care. Silver Spring, Md: IEEE Computer Society Press; 1983. 7 664-7.
16 Kaplan B. Barriers to medical computing: history, diagnosis, and therapy for the medical computing ‘lag’. In Ackerman MJ. editor Proc, Symp Computer Applications in Medical Care. Silver Spring, Md: IEEE Computer Society Press; 1985. 9 400-4.
17 Benson T. Why British GPs use computers and hospital doctors do not. Proc AMIA Symp. 2001: 42-6.
18 Kaplan B. Evaluating informatics applications-clinical decision support systems literature review. Int J Med Inform 2001; 64: 15-37.
19 Kaplan B. Evaluating informatics applications-some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 2001; 64: 39-56.
20 Ash JS, Gorman PN, Hersh WR. Physician order entry in U.S. hospitals. Proc AMIA Symp 1998; 235-9.
21 Ash JS, Gorman PN, Lavelle M, Lyman J, Fournier L, Carpenter J. et al. Physician order entry: results of a cross-site case study. Methods Inf Med. Submitted.
22 Haruki Y, Ogushi Y, Okada Y, Kimura M, Kumamoto I, Sekita Y. Status and perspective of hospital information systems in Japan. Methods Inf Med 1999; 38 (03) 200-6.
23 Campbell JR, Givner N, Seelig CB, Greer AL, Patil K, Wigton RS. et al. Computerized medical records and clinic function. MD Computing 1989; 6 (05) 282-7.
24 Kaplan B. Reducing barriers to physician data entry for computer-based patient records. Tops Health Inform Management 1994; Aug 15 (01) 24-34.
25 Shu K, Boyle D, Spurr C, Horsky J, Heiman H, O’Connor P, Lepore J, Bates D. Comparison of time spent writing orders on paper with computerized physician order entry. In Patel V, Rogers R, Haux R. editors Medinfo 2001. Amsterdam: IOS Press; 2001: 1207-11.
26 Weiner M, Gress T, Thiemann DR, Jenckes M, Reel SL, Mandell SF, Bass EB. Contrasting views of physicians and nurses about an inpatient computer-based provider order-entry system. J Am Med Inform Assoc 1999; May/June 6 (03) 234-44.
27 Ash JS, Lyman J, Carpenter J, Fournier L. A diffusion of innovations model of physician order entry. Proc AMIA Symp 2001; 22-6.
28 Ash JS, Gorman PN, Hersh WR, Poulsen SP. Perceptions of house officers who use physician order entry. Proc AMIA Symp 1999; 471-5.
29 Ash JS, Gorman PN, Lavelle M, Lyman J. Multiple perspectives on physician order entry. Proc AMIA Symp 2000; 27-31.
30 Ash J, Gorman P, Lavelle M, Lyman J, Fournier L. Investigating physician order entry in the field: lessons learned in a multi-centered study. In Patel V, Rogers R, Haux R. editors Medinfo 2001. Amsterdam: IOS Press; 2001: 1107-11.
31 Anderson JG, Jay SJ. editors Use and impact of computers in clinical medicine. New York: Springer; 1987
32 Protti DJ, Haskell AR. Managing information in hospitals;60% social, 40% technical. Proc IMIA Working Conference on Trends in Hospital Information Systems. North Holland Publishing; 1992: 45-8.
33 Lorenzi NM, Riley RT, Blyth AJC, Southon G, Dixon BJ. Antecedents of the people and organizational aspects of medical informatics: review of the literature. J Am Med Inform Assoc 1997; Mar-Apr 4 (02) 79-93.
34 Anderson JG, Aydin CE, Jay SJ. editors Evaluating health care information systems: methods and applications. Thousand Oaks, CA: Sage Publications; 1994
35 Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York: Springer; 1997
36 van Gennip EMS, Talmon J. editors Assessment and evaluation of information technologies. Studies in Health Technology and Informatics, vol 17. Amsterdam: IOS Press; 1995
37 Rigby M. Evaluation:16 powerful reasons why not to do it-and 6 over-riding imperatives. In Patel V, Rogers R, Haux R. editors Medinfo 2001. Amsterdam: IOS Press; 2001: 1198-201.
38 Kaplan B. Addressing organizational issues into the evaluation of medical systems. J Am Med Inform Assoc 1997; March/April 4 (02) 94-101.
39 Fischer PJ, Stratman WC, Lundsgaarde HP, Steele DJ. User reaction to PROMIS: issues related to acceptability of medical innovations. In O’Neill JT. editor Proc Symp Computer Applications in Medical Care Silver Spring. IEEE Computer Society Press 1980; 4: 1722-30.
39a Reprinted. in Anderson JG, Jay SJ. editors Use and impact of computers in clinical medicine. New York: Springer; 1987: 284-301.
40 Lundsgaarde HP, Fischer PJ, Steele DJ. Human problems in computerized medicine. Publications in Anthropology, 13 Lawrence, KS: The University of Kansas; 1981
41 Siegel C, Alexander MJ, Dlugacz YD, Fischer S. Evaluation of a computerized drug review system: impact, attitudes, and interactions. Comp Biomed Res 1984; 17: 19-435.
42a Reprinted in. Anderson JG, Jay SJ. editors Use and impact of computers in clinical medicine. New York: Springer; 1987: 238-66.
42 Dlugacz YD, Siegel C, Fischer S. Receptivity toward uses of a new computer application in medicine. In Blum BI. editor Proc Symp Computer Applications in Medical Care Silver. Spring: IEEE Computer Society Press; 1980. 6 384-91.
43 Anderson JG, Jay SL. Computers and clinical judgment: the role of physician networks. Soc Sci Med 1985; 20: 967-79.
44a Reprinted in. Anderson JG, Jay SJ. editors Use and impact of computers in clinical medicine. New York: Springer; 1987: 161-84.
44 Anderson JG, Jay SL, Schwerr HM, Anderson MM, Kassing D. Physician communication networks and the adoption and utilization of computer applications in medicine. In Anderson JG, Jay SJ. editors Use and impact of computers in clinical medicine. New York: Springer; 1987: 185-99.
45 Barrett JP, Barnum RA. et al. Evaluation of a medical information system in a community hospital. Battelle Columbus Laboratories (NTIS PB 248 340); Dec. 19. 1975
46 Kaplan BM. Computers in medicine 1950-1980: the relationship between history and policy. Doctoral Dissertation: University of Chicago; 1983
47 Collen MF. A history of medical informatics in the United States:1950 to 1990. Bethesda, MD: American Medical Informatics Association; 1995
48 Farlee C, Goldstein B. Hospital organization and computer technology: the challenge of change. New Brunswick, NJ: Health Care Systems Research; 1972
49 Brown B, Harbort B, Kaplan B, Maxwell J. Guidelines for managing the implementation of automated medical systems. In Heffernan HG. editor Proc. Symp Computer Applications in Medical Care. Silver Spring: IEEE Computer Society Press; 1981. 5 791-6.
50 Brown B, Harbort B, Kaplan B. Management issues in automating medical systems. J Clinical Eng 1983; Jan-Mar 8 (01) 23-30.
51 Kaplan B. Pre-installation strategies: setting the social stage for user acceptance. National Report on Computers and Health. 1983. Jan;28: special report
52 Sigurdardottir H, Skov D, Bartholdy C, Wann-Larsen S. Evaluating the usefulness of monitoring change readiness in organisations which plan on implementing health informatics systems. In: IT in Health Care Sociotechnical Approaches. International Conference Proceedings, 6-7, September. The Netherlands: Erasmus University Rotterdam; 2001: 52
53 Hanmer JC. Diffusion of medical technologies: comparison with ADP systems in the medical environment. In O’Neill JT. editor Proc Symp Computer Applications in Medical Care Silver. Spring: IEEE Computer Society Press; 1980. 4 1731-6.
54 Lorenzi N, Riley R T. Managing change: an overview. J Am Med Inform Assoc 2000; Mar-Apr 7 (02) 116-24.
55 Ash J. Managing change: analysis of a hypothetical case. J Am Med Inform Assoc 2000; Mar-Apr 7 (02) 125-34.
56 Department of Health, NHS Executive. Information Management Group. Project review: objective evaluation. Guidance for NHS managers on evaluating information systems projects. Department of Health, NHS Executive, 1996. IMG D 4042.
57 Coiera E. When conversation is better than computation. J Am Med Inform Assoc 2000; 7 (03) 277-86.
58 Kaplan B. Information technology and three studies of clinical work. ACM SIGBIO Newsletter 1995; May 15 (02) 2-5.
59 Safran C, Jones PC, Rind D, Booker B, Cytryn KN, Patel VL. Electronic communication and collaboration in a health care practice. Artif Intell Med 1998; Feb 12 (02) 137-51.
60 Sicotte C, Denis JL, Lehoux P. The computer based patient record: a strategic issue in process innovation. J Med Syst 1998; Dec 22 (06) 431-43.
61 Mitev N, Kerkham S. Organization and implementation issues of patient data management systems in an intensive care unit. J End User Computing 2001; July-Sep 13 (03) 20-9.
62 Grémy F, Fessler JM, Bonnin M. Information systems evaluation and subjectivity. Int J Med Inform 1999; 56 1-3 13-23.
63 Kaplan B. Development and acceptance of medical information systems: an historical overview. J Health Human Resources Admin 1988; Summer 11 (01) 9-29.
64 Aydin C. Computerized order entry in a largemedicalcenter: evaluatinginteractions between departments. In Anderson JG, Aydin CE, Jay SJ. editors Evaluating health care information systems: Approaches and applications. Thousand Oaks, Cal: Sage; 1994: 260-75.
65 Massaro TA. Introducing physician order entry at a major academic medical center:1. Impact on organizational culture and behavior. Acad Med. 1993; 68 (01) 20-5.
66 Patel VL, Allen VG, Arocha JF, Shortliffe EH. Representing clinical guidelines in GLIF: individual and collaborative expertise. J Am Med Inform Assoc 1998; Sep-Oct 5 (05) 467-83.
67 Harrop VM. Virtual healthcare delivery: defined, modeled, and predictive barriers to implementation identified. Proc AMIA Symp 2001; 244-8.
68 May CR, Gask L, Atkinson T, Ellis NT, Mair F, Esmail A. Resisting and promoting new technologies in clinical practice: the case of ‘telepsychiatry. Soc Sci Med 2001; 52 (12) 1889-901.
69 Kaplan B, Duchon D. Combining qualitative and quantitative approaches in information systems research: a case study. Manag Inform Sys Q 1998; December 12 (04) 571-86.
70 Nordyke RA, Kulikowski CA. An informatics-based chronic disease practice: Case study of a 35-year computer-based longitudinal record system. J Am Med Inform Assoc 1998; Jan./Feb 5 (01) 88-103.
71 Aarts J, Peel V, Wright G. Organizational issues in health informatics: a model approach. Int J Med Inform 1998; Oct-Dec 52 1-3 235-42.
72 Folz-Murphy N, Partin M, Williams L, Harris CM, Lauer MS. Physician use of an ambulatory medical record system: matching form and function. Proc AMIA Symp 1998; 260-4.
73 Clarke K, O’Moore R, Smeets R, Talmon J, Brender J, McNair P. et al. A methodology for evaluation of knowledge-based systems in medicine. Artif Intell Med 1994; Apr 6 (02) 107-21.
74 Dixon DR. The behavioral side of information technology. Int J Med Inform 1999; Dec 56 1-3 117-23.
75 Southon G, Sauer C, Dampney K. Lessons from a failed information systems initiative: issues for complex organizations. Int J Med Inform 1999; Jul 55 (01) 33-46.
76 Heathfield HA, Wyatt J. Philosophies for the design and development of clinical decision-support systems. Methods Inf Med 1993; 32: 1-8.
77 Kaplan B. The computer prescription: medical computing, public policy, and views of history. Science, Technology, and Human Values 1995; Winter 20 (01) 5-38.
78 Demeester M. Cultural aspects of information technology implementation. Int J Med Inform 1999; Dec 56 1-3 25-41.
79 Forsythe DE. New bottles, old wine: hidden cultural assumptions in a computerized explanation system for migraine sufferers. Med Anthropology Q 1996; 10 (04) 551-74.
80 Whitley EA, Pouloudi A. Studying the translations of NHS net. J End User Computing 2001; July-Sep 13 (03) 31-40.
81 Covvey HD. When is a system only a system?. Healthcare Inform Management & Comm Canada 2001; 15 (01) 46-7.
82 Graves II W, Nyce JM. Normative models and situated practice in medicine. Inform Decision Technologies 1992; 18: 143-9.
83 Nyce JM, Graves III W. The construction of knowledge in neurology: implications for hypermedia system development. Artif Intell Med 1990; Dec 292: 315-22.
84 Nyce JM, Timpka T. Work, knowledge and argument in specialist consultations: incorporating tacit knowledge into system design and development. Med Biol Eng Comput 1993; Jan 31 (01) HTA16-HTA 19
85 Timpka T, Rauch E, Nyce JM. Towards productive Knowledge-Based Systems in clinical organizations: a methods perspective. Artif Intell Med 1994; Dec 6 (06) 501-19.
86 Gadd CS, Baskaran P, Lobach DF. Identification of design features to enhance utilization and acceptance of systems for Internet-based decision support at the point of care. Proc AMIA Symp 1998; 91-5.
87 Gamm LD, Barsukiewicz CK, Dansky KH, Vasey JJ, Bisordi JE, Thompson PC. Pre-and post-control model research on end-users’ satisfaction with an electronic medical record: preliminary results. Proc AMIA Symp 1998; 225-9.
88 Kaplan B, Morelli R, Goethe J. Preliminary findings from an evaluation of the acceptability of an expert system in psychiatry. Extended Proc AMIA Symp. 1997. (CD-ROM)
89 Karlsson D, Ekdahl C, Wigertz O, Forsum U. A qualitative study of clinicians ways of using a decision-support system. Proc AMIA Symp 1997; 268-72.
90 Moore LA. Reaction of medical personnel to a medical expert system for stroke. In Anderson JG, Aydin CE, Jay SJ. editors Evaluating health care information systems: Approaches and applications. Thousand Oaks, Cal: Sage; 1994: 226-44.
91 Grigsby J. You got an attitude problem, or what?. Telemed Today. 1995: 32-4.
92 Burghgraeve P, De Maeseneer J. Improved methods for assessing information technology in primary health care and an example from telemedicine. J Telemed Telecare 1995; 1: 157-64.
93 Wyatt JC. Commentary: telemedicine trials-clinical pull or technology push?. BMJ 1996; 313: 380-1.
94 Perednia DM. Telemedicine system evaluation, transaction models, and multicentred research. J Am Health Inform Manag Assoc 1996; 67 (01) 60-3.
95 Grigsby J, Schlenker R, Kaehny M, Shaughnessy P. Analytic framework for evaluation of telemedicine. Telemed J 1995; 1 (01) 31-9.
96 Grigsby J. Sentenced to life without parole doing outcomes research. Telemed Today. 1996; 40-1.
97 McIntosh E, Cairns J. A framework for the economic evaluation of telemedicine. J Telemed Telecare 1997; 3: 132-9.
98 Yellowlees P. Practical evaluation of telemedicine systems in the real world. Tele Med 1997; 36: 90-3. also J Telemed Telecare 1998;4 Suppl 1:56-7
99 Bashshur R. On the definition and evaluation of telemedicine. Telemed J 1995; 1 (01) 19-30.
100 Nykanen P, Chowdhury S, Wigertz O. Evaluation of decision support systems in medicine. Comp Meths Programs Biomed 1991; 34: 229-38.
101 Filiberti D, Wallace J, Koteeswaran R, Neft D. A telemedicine transaction model. Telemed J 1995; 1 (03) 237-47.
102 DeChant H, Tohme W, Mun S, Hayes W. Health systems evaluation of telemedicine: a staged approach. Telemed J 1996; 2 (04) 303-12.
103 Tohme WG, Hayes WS, Dai H, Komo D. Evaluation of a telemedicine platform for three medical applications Comp Assisted Rad. 1996: 505-11.
104 Allen D. Telemental health services today. Telemed Today 1994; 2 (02) 1-24.
105 Klecun-Dabrowska E, Cornford T. Evaluating telehealth: the search for an ethical perspective. Cambridge Q Healthcare Ethics 2001; 10 (02) 161-9.
106 Kaplan B, Brennan PF. Consumer informatics supporting patients as coproducers of quality. J Am Med Inform Assoc 2001; Jul-Aug 8 (04) 309-16.
107 Reiser SJ. Medicine and the reign of technology. Cambridge: Cambridge University Press; 1978
108 Bosk C, Frader J. The impact of place of decision-making on medical decisions. In O’Neill JT. editor Proc Symp Computer Applications in Medical Care Silver. Spring: IEEE Computer Society Press; 1980. 4 1326-9.
109 Qavi T, Corley L, Kay S. Nursing staff requirements for telemedicine in the neonatal intensive care unit. J End User Computing 2001; July-Sep 13 (03) 5-13.
110 Aanestad M, Edwin B, Måarvik R. Image quality as a sociotechnical network. In IT in Health Care Sociotechnical Approaches. International Conference proceedings,6-7. September The Netherlands: Erasmus University Rotterdam; 2001: 16
111 Kaplan B. Objectification and negotiation in interpreting clinical images: implications for computer-based patient records. Artif Intell Med 1995; 7: 439-54.
112 Kaplan B, Farzanfar R, Friedman RH. Ethnographic interviews to elicit patients’ reactions to an intelligent interactive telephone health behavior advisor system. Proc AMIA Symp 1999; 555-9.
113 Travers DA, Downs SM. Comparing the user acceptance of a computer system in two pediatric offices: a qualitative study. Proc AMIA Symp 2000; 853-7.
114 Kaplan B. Computer Rorschach Test: what do you see when you look at a computer?. Physicians & Comput 2001; Jan 18 (05) 12-3.
115 Nøhr C. The evaluation of expert diagnostic systems-How to assess outcomes and quality parameters?. Artif Intell Med 1994; Apr 6 (02) 123-35.
116 Grémy F, Bonnin M. Evaluation of automatic health information systems. What and how? In. van Gennip EMS, Talmon JL. editors Assessment and evaluation of information technologies, Studies in Health Technology and Informatics, vol 17. Amsterdam: IOS Press; 1995: 9-20.
117 Jørgensen T. Measuring effects. In van Gennip EMS, Talmon JL. Editors Assessment and evaluation of information technologies, Studies in Health Technology and Informatics, vol 17. Amsterdam: IOS Press; 1995: 99-109.
118 Miller RA. Medical diagnostic decision support systems-past, present, and future: a threaded bibliography and brief commentary. J Am Med Inform Assoc 1994; 1: 8-27.
119 Heathfield HA, Buchan IE. Current evaluations of information technology in health care are often inadequate. BMJ October 19 1996; 313 07063 1008
120 Dambro MR, Weiss BD, McClure C, Vuturo AF. An unsuccessful experience with computerized medical records in an academic medical center. J Med Ed 1988; 63: 617-23.
121 Chessare JB, Torok KE. Implementation of COSTAR in an acaedmic group practice of general pediatrics. MD Computing 1993; Jan-Feb 10 (01) 23-7.
122 Sicotte C, Denis JL, Lehoux P, Champagne F. The computer-based patient record: challenges towards timeless and spaceless medical practice. Med Syst 1998; Aug 22 (04) 237-56.
123 Tonnesen AS, LeMaistre A, Tucker D. Electronic medical record implementation barriers encountered during implementation. In: Proc AMIA Symp. 1999: 625-66.
124 Balka E. Getting the big picture: The macro-politics of information system development (and failure) in a Canadian hospital. In. IT in Health Care Sociotechnical Approaches. International Conference proceedings, 6-7 September. The Netherlands: Erasmus University Rotterdam; 2001: 17
125 Jeffcott M. Technology alone will never work: Understanding how organisational issues contribute to user neglect and information systems failure in healthcare IT. In. Health Care Sociotechnical Approaches. International Conference Proceedings, 6-7 September. The Netherlands: Erasmus University Rotterdam; 2001: 35
126 Zuiderent T. Changing care and theory: a theoretical joyride as the safest journey. In. IT in Health Care Sociotechnical Approaches. International Conference Proceedings, 6-7 September. The Netherlands: Erasmus University Rotterdam; 2001: 62
127 May C, Ellis NT. When protocols fail: technical evaluation, biomedical knowledge, and the social production of ‘facts’ about a telemedicine clinic. Soc Sci Med 2001; Oct 53 (08) 989-1002.
128 Jones MR. An interpretive method for the formative evaluation of an electronic patient record system. In Remenyi D, Brown A. editors Procs of the 8th European Conference on IT Evaluation. Oxford: 17-18 September 2001: 349-56.
129 Beynon-Davies P. The London Ambulance Service’s computerised dispatch system: a case study in information system failure. University of Glamorgan Pontyprid. 1993
130 Beynon-Davies P. Information systems ‘failure’: the case of the London Ambulance Service’s Computer Aided Despatch project. European J Inform Syst 1995; 4: 171-84.
131 Silva L, Backhouse J. Becoming part of the furniture: the institutionalization of information systems. In Lee AS, Leibenau J, DeGross JI. Information Systems and Qualitative Research. London: Chapman & Hall; 1997: 389-414.
132 Friedman CP, Wyatt JC. Publication bias in medical informatics. J Am Med Inform Assoc 2001; Mar/Apr 8 (02) 189-91.
133 Heathfield HA, Peel V, Hudson P, Kay S, Mackay L, Marley T. et al. Evaluating large scale health information systems: from practice towards theory. Proc AMIA Symp 1997; 116-20.
134 Penrod LE, Gadd CS. Attitudes of academic-based and community-based physicians regarding EMR use during outpatient encounters. Proc AMIA Symp 2001; 528-32.
135 Barley SR. Technology as an occasion for structuring: evidence from observations of CT scanners and the social order of radiology departments. Adm Sci Q 1986; 31: 78-108.
136 Isaacs S. The power distance between users of information technology and experts and satisfaction with the information system: implications for crosscultural transfer of IT. In Patel V, Rogers R, Haux R. editors Medinfo 2001. Amsterdam: IOS Press; 2001: 1155-7.
137 van Gennip EMS, Talmon JL. editors Introduction. In van Gennip EMS, Talmon JL. editors Assessment and evaluation of information technologies, Studies in Health Technology and Informatics, vol 17. Amsterdam: IOS Press; 1995: 1-5.
138 van der Loo RP. Overview of published assessment and evaluation studies. In van Gennip EMS, Talmon JL. editors Assessment and evaluation of information technologies, Studies in Health Technology and Informatics, vol 17. Amsterdam: IOS Press; 1995: 261-82.
139 Tierney WM, Overhage JM, McDonald CJ. An plea for controlled trials in medical informatics. J Am Med Inform Assoc 1994; July/Aug 1 (04) 353-5.
140 Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998; June 27 316 07149 1959-61.
141 Forsythe D, Buchanan B. Broadening our approach to evaluating medical expert systems. In Clayton PD. editor Symposium on computer applications in medical care. McGraw-Hill, New York: 1991: 8-12.
142 Ellis NT, Aarts J, Kaplan B, Leonard K. Alternatives to the controlled trial: new paradigms for evaluation. Los Angeles, CA: AMIA Symp; 2000
143 Kaplan B, Aarts J, Hebert M, Klecun-Dabrowska E, Lewis D, Vimarlund V. New approaches to evaluation: alternatives to the randomized controlled trial-qualitative approaches to design and evaluation: theory and practice. In Patel V, Rogers R, Haux R. editors Medinfo 2001. Amsterdam: IOS Press; 2001: 1531
144 Kaplan B, Atkinson C, Ellis N, Jones M, Klecun-Dabrowska E, Teasdale S. et al. Evaluation in the UK’s National Health Service. In Patel V, Rogers R, Haux R. editors Medinfo 2001. Amsterdam: IOS Press; 2001. 1529
145 Kaplan B, Anderson JG, Jaffe C, Leonard K, Zitner D. New approaches to evaluation: alternatives to the randomized controlled trial-quantitative models for evaluation. In Patel V, Rogers R, Haux R. editors Medinfo 2001. >Amsterdam: IOS Press; 2001. 1456
146 Kaplan B, Maxwell JA. Qualitative research methods for evaluating computer information systems. In Anderson JG, Aydin CE, Jay SJ. editors Evaluating health care information systems: approaches and applications. Thousand Oaks, Cal: Sage; 1994: 45-68.
147 Talmon J, Enning J, Castãneda G, Eurlings F, Hoyer D, Nykänen P. et al. The VATAM guidelines. Int J Med Inform 1999; Dec 56 1-3 107-15.
148 Shaw NT. CHEATS: A generic evaluation framework for information communication technologies: invited paper. Comp Meths Programs Biomed In press. 2001
149 Gardner RM, Lundsgaarde HP. Assessment of user acceptance of a clinical expert system. J Am Med Inform Assoc 1994; Nov-Dec 1 (06) 428-38.
150 Saranummi N. Supporting system development with technology assessment. In van Gennip EMS, Talmon JL. eds Assessment and evaluation of information technologies, Studies in Health Technology and Informatics, vol 17. >Amsterdam: IOS Press; 1995: 35-44.
151 Faber M. User involvement in design and introduction of a Electronic Patient Record. On the never predicatble interplay between technical objects and social organizations. In IT in Health Care Sociotechnical Approaches. International Conference Proceedings 6-7. September 2001. The Netherlands: Erasmus University Rotterdam; 27
152 Kaplan B. Fitting system design to work practice: using observation in evaluating a clinical imaging system. In Ahuja MK, Galletta DF, Watson HJ. editors Proc: Americas Conference on Information Systems, Volume IV: Information Systems—Collaboration Systems and Technology, and Organizational Systems and Technology. Pittsburgh: Association for Information Systems; 1995: 86-8.
153 Lau F, Hayward R. Building a virtual network in a community health research training program. J Am Med Inform Assoc 2000; Jul/Aug 7 (04) 361-77.
154 Timpka T, Arborelius E. The GP’s dilemmas: a study of knowledge need and use during health care consultations. Meths Inform Med 1990; Jan 29 (01) 23-9.
155 Weaver RR. Evaluating the Problem Knowledge Coupler: a case study. In Anderson JG, Aydin CE, Jay SJ. editors Evaluating health care information systems: approaches and applications. Thousand Oaks, Cal: Sage; 1994: 203-25.
156 Schubart JR, Einbinder JS. Evaluation of a data warehouse in an academic health sciences center. Int J of Med Inform 2000; 60: 319-33.
157 Anderson JG, Aydin CE, Kaplan B. An analytical framework for measuring the effectiveness/impacts of computer-based patient record systems. In Nunamaker JF, Sprague Jr RH. editors Proc: Twenty-Eighth Hawaii International Conference on Systems Science, Vol IV: information systems – collaboration systems and technology, organizational systems and technology. Los Alamitos, Cal: IEEE Computer Society Press; 1995: 767-76.
158 Aarts J. On articulation and localization-some sociotechnical issues of design, implementation, and evaluation of knowledge based systems. In Quaglini S, Barahona P, Andreassen S. editors Artificial Intelligence in Medicine, Proc of the 8th Conference on Artificial Intelligence in Medicine in Europe, AIME 2001. Berlin: Springer; 2001: 16-9.
159 Berg M. Patient care information systems and health care work: a sociotechnical approach. Int J Med Inform 1999; Aug 55 (02) 87-101.
160 Berg M, Gorman E. The contextual nature of medical information. Int J Med Inform 1999; Dec 56 1-3 51-60.
161 Berg M, Langenberg C, Berg I v.d, Kwakkernaat J. Considerations for sociotechnical design: experiences with an electronic patient record in a clinical context. Int J Med Inform 1998; Oct/Dec 52 1-3 243-51.
162 Stoop A, Berg M. Health care ICT evaluation: state of the art in a dynamic field. In IT in Health Care Sociotechnical Approaches. International Conference Proceedings, 6-7. September The Netherlands: Erasmus University Rotterdam; 2001: 55
163 Reddy MC. Sociotechnical requirements for healthcare systems. In IT in Health Care Sociotechnical Approaches. International Conference Proceedings, 6-7 September. 2001. The Netherlands: Erasmus University Rotterdam; 49
164 Bygholm A. Activity thepry as a framework for conducting end-use support. In IT in Health Care Sociotechnical Approaches. International Conference Proceedings, 6-7 September. 2001. The Netherlands: Erasmus University Rotterdam; 21
165 Kristensen M. Assessing a prerequisite for a constructivist approach – change readiness in organisations implementing health informatics systems. In IT in Health Care Sociotechnical Approaches. International Conference Proceedings, 6-7 September. 2001. The Netherlands: Erasmus University Rotterdam; 38
166 IT in Health Care Sociotechnical Approaches. International Conference Proceedings, 6-7 September. The Netherlands: Erasmus University Rotterdam; 2001
167 Suchman L. Representations of work. Commun ACM 1995; 38 (09) 33-4.
168 Suchman LA. Plans and situated actions: the problem of human-machine communication. Cambridge: Cambridge University Press; 1987
169 Butler KA, Esposito C, Hebron R. Connecting the design of software to the design of work. Commun of the ACM 1999; 42 (01) 38-46.
170 Musen MA. Architectures for architects. Methods Inf Med 1993; 32: 12-3.
171 Artificial Intelligence in Medicine. 1995. Oct 7
172 Greenbaum J, Kyng M. Design at work: cooperative design of computer systems. Hillsdale NJ: Lawrence Erlbaum Associates; 1991
173 Winograd T, Flores F. Understanding computers and cognition: a new foundation for design. Norwood NJ: Ablex; 1986
174 Sjöberg C, Timpka T. Participatory design of information systems in health care. J Am Med Inform Assoc 1998; Mar-Apr 5 (02) 177-83.
175 Timpka T, Hedblom P, Holmgren J. Action design: using an object-oriented environment for group process development of medical software. In Timmers T. editor Software engineering in medical informatics. Amsterdam: Elsevier; 1991: 151-65.
176 May CR, Mort M, Mair F, Ellis NT, Gask L. Evaluation of new technologies in health care systems: what’s the context?. Health Inform J 2000; 6 (02) 67-70.
177 Williams TR, May CR, Mair F, Mort M, Shaw N, Gask L. Normative models of health technology assessment and the social production of evidence about telehealth care. In review. 2001
178 May C, Gask L, Ellis NT, Atkinson T, Mair F, Smith C. et al. Telepsychiatry evaluation in the north west of England: preliminary results. Telemed Telecare 2000; 6 (Suppl. 01) Supplement 20-2.
179 Yoxen E. Seeing with sound: a study of the development of medical images. In Bijker WE, Hughes T, Pinch T. editors The social construction of technological systems. Cambridge, Mass: MIT Press; 1987: 281-306.
180 May M, Mort M, Williams T, Mair F, Shaw NT. Understanding the evaluation of telemedicine: The play of the social and the technical, and the shifting sands of reliable knowledge. In IT in Health Care Sociotechnical Approaches. International Conference Proceedings, 6-7 September. The Netherlands: Erasmus University Rotterdam; 2001: 43
181 Aarts J, Peel V. Using a descriptive model of change when implementing largescale clinical information systems to identify priorities for future research. Int J Med Inform 1999; Dec 56 1-3 43-50.
182 Aarts J, Peel V, Wright G. Organizational issues in health informatics: a model approach. Int J Med Inform 1998; Oct-Dec 52 1-3 235-42.
183 Donabedian A. The quality of medical care: methods for assessing and monitoring the quality of care for research and for quality assurance programs. Science 1978; 200: 856-64.
184 Donabedian A. Explorations in quality assessment and monitoring. Vol I: The defninition of quality and approaches to its assessment. Ann Arbor: Health Administration Press; 1980
185 Hebert M. Telehealth success: evaluation framework development. In Patel V, Rogers R, Haux R. editors Medinfo 2001. >Amsterdam: IOS Press; 2001: 1145-9.
186 Cornford T, Doukidis I G, Forster D. Experience with a structure, process and outcome framework for evaluating an information system. Omega, Int J of Manage Sci 1994; 255 (05) 491-504.
187 Talmon JL. Workshop WG15: technology assessment and quality improvement. In Patel V, Rogers R, Haux R. editors Medinfo 2001. Amsterdam: IOS Press; 2001. 1540