Download PDF
Yearb Med Inform 2017; 26(01): 59-67
DOI: 10.15265/IY-2017-010
Special Section: Learning from Experience: Secondary Use of Patient Data
Working Group Contributions
Georg Thieme Verlag KG Stuttgart

Evaluation Considerations for Secondary Uses of Clinical Data: Principles for an Evidence-based Approach to Policy and Implementation of Secondary Analysis

A Position Paper from the IMIA Technology Assessment & Quality Development in Health Informatics Working Group
P. J. Scott
1   University of Portsmouth, Centre for Healthcare Modelling and Informatics, Portsmouth, United Kingdom
,
M. Rigby
2   Keele University, School of Social Science and Public Policy, Keele, United Kingdom
,
E. Ammenwerth
3   UMIT, University for Health Sciences, Medical Informatics and Technology, Institute of Medical Informatics, Hall in Tyrol, Austria
,
J. Brender McNair
4   Aalborg University, Department of Health Science & Technology, Aalborg, Denmark
,
A. Georgiou
5   Macquarie University, Australian Institute of Health Innovation, Sydney, Australia
,
H. Hyppönen
6   National Institute for Health and Welfare, Information Department, Helsinki, Finland
,
N. de Keizer
7   Academic Medical Center, Department of Medical Informatics, Amsterdam, The Netherlands
,
F. Magrabi
5   Macquarie University, Australian Institute of Health Innovation, Sydney, Australia
,
P. Nykänen
8   University of Tampere, School of Information Sciences, Tampere, Finland
,
W. T. Gude
7   Academic Medical Center, Department of Medical Informatics, Amsterdam, The Netherlands
,
W. Hackl
3   UMIT, University for Health Sciences, Medical Informatics and Technology, Institute of Medical Informatics, Hall in Tyrol, Austria
› Author Affiliations
Further Information

Publication History

Publication Date:
11 September 2017 (online)

   Permissions and Reprints

Summary

Objectives: To set the scientific context and then suggest principles for an evidence-based approach to secondary uses of clinical data, covering both evaluation of the secondary uses of data and evaluation of health systems and services based upon secondary uses of data.

Method: Working Group review of selected literature and policy approaches.

Results: We present important considerations in the evaluation of secondary uses of clinical data from the angles of governance and trust, theory, semantics, and policy. We make the case for a multi-level and multi-factorial approach to the evaluation of secondary uses of clinical data and describe a methodological framework for best practice. We emphasise the importance of evaluating the governance of secondary uses of health data in maintaining trust, which is essential for such uses. We also offer examples of the re-use of routine health data to demonstrate how it can support evaluation of clinical performance and optimize health IT system design.

Conclusions: Great expectations are resting upon “Big Data” and innovative analytics. However, to build and maintain public trust, improve data reliability, and assure the validity of analytic inferences, there must be independent and transparent evaluation. A mature and evidence-based approach needs not merely data science, but must be guided by the broader concerns of applied health informatics.

Keywords

Data mining - secondary use - informed consent - evidence-based practice - program evaluation
 

  • References

  • 1 OECD. Health Data Governance: Privacy, Monitoring and Research. Paris: OECD Publishing; 2015 [cited 2016 21 November]. Available from: http://www.oecd-ilibrary.org/social-issues-migration-health/health-data-governance_9789264244566-en
    PubMedGoogle Scholar
  • 2 Safran C, Bloomrosen M, Hammond WE, Labkoff S, Markel-Fox S, Tang PC. et al. Toward a national framework for the secondary use of health data: an American Medical Informatics Association White Paper. J Am Med Inform Assoc 2007; Jan-Feb; 14 (01) 1-9.
    CrossrefPubMedGoogle Scholar
  • 3 OECD. ICTs and the Health Sector: Towards Smarter Health and Wellness Models. Paris: OECD Publishing; 2013 [cited 2016 21 November]. Available from: http://www.oecd.org/sti/ieconomy/ ict-and-the-health-sector.htm
    PubMedGoogle Scholar
  • 4 Rigby M, Ronchi E, Graham S. Evidence for building a smarter health and wellness future--key messages and collected visions from a joint OECD and NSF workshop. Int J Med Inform 2013; Apr; 82 (04) 209-19.
    PubMedGoogle Scholar
  • 5 Friedman C, Rigby M. Conceptualising and creating a global learning health system. Int J Med Inform 2013; Apr; 82 (04) e63-71.
    PubMedGoogle Scholar
  • 6 Wyatt JC. Evidence-based Health Informatics and the Scientific Development of the Field. In: Ammenwerth E, Rigby M. editors. Evidence-based Health Informatics: Promoting safety and efficiency through scientific methods and ethical policy. Stud Health Technol Inform Amsterdam. IOS Press; 2016: 14-24.
    Google Scholar
  • 7 Bowling A. Research methods in health: investigating health and health services. 4th edition. Buckingham: McGraw Hill: Open University Press; 2014
    Google Scholar
  • 8 Campos-Castillo C, Anthony DL. The double-edged sword of electronic health records: implications for patient disclosure. J Am Med Inform Assoc 22 (e1) e130-40.
    PubMedGoogle Scholar
  • 9 Open Knowledge Network. The Open Definition [cited 2017 12 April]. Available from: http://open-definition.org/
    PubMedGoogle Scholar
  • 10 Verhulst S, Noveck BS, Caplan R, Brown K, Paz C. The Open Data Era in Health and Social Care. 2014 [cited 2017 12 April]. Available from: http://thegovlab.org/nhs/
    PubMedGoogle Scholar
  • 11 Sanderson SC, Brothers KB, Mercaldo ND, Clayton EW, Antommaria AH, Aufox SA. et al. Public Attitudes toward Consent and Data Sharing in Bio-bank Research: A Large Multi-site Experimental Survey in the US. Am J Hum Genet 2017; Mar 02; 100 (03) 414-27.
    PubMedGoogle Scholar
  • 12 Patil S, Lu H, Saunders CL, Potoglou D, Robinson N. Public preferences for electronic health data storage, access, and sharing — evidence from a pan-European survey. J Am Med Inform Assoc 2016; 23 (06) 1096-106.
    PubMedGoogle Scholar
  • 13 El Emam K, Jonker E, Arbuckle L, Malin B. A systematic review of re-identification attacks on health data. PLoS One 2011; 06 (12) e28071.
    PubMedGoogle Scholar
  • 14 Information Commissioner’s Office. Anonymisation: managing data protection risk code of practice. 2012 [cited 2017 12 April]. Available from https://ico.org.uk/media/1061/anonymisation-code.pdf
    PubMedGoogle Scholar
  • 15 NHS National Services Scotland. About ISD: Confidentiality. 2010 [cited 2016 1 December]. Available from: http://www.isdscotland.org/About-ISD/Confidentiality/#secondaryUse
    PubMedGoogle Scholar
  • 16 WebFinance Inc. Governance. 2016 [cited 2016 21 November]. Available from: http://www.business-dictionary.com/definition/governance.html
    PubMedGoogle Scholar
  • 17 The Data Governance Institute. Defining data governance. 2015 [cited 2016 21 November]. Available from: http://www.datagovernance.com/defining-data-governance/
    PubMedGoogle Scholar
  • 18 Ingenerf J. Telemedicine and terminology: different needs of context information. IEEE Trans Inf Technol Biomed 1999; Jun; 03 (02) 92-100.
    PubMedGoogle Scholar
  • 19 Coiera E. Guide to Health Informatics. 3rd ed. CRC press; 2015
    Google Scholar
  • 20 Smith SW, Koppel R. Healthcare information technology’s relativity problems: a typology of how patients’ physical reality, clinicians’ mental models, and healthcare information technology differ. J Am Med Inform Assoc 2014; 21 (01) 117-31.
    PubMedGoogle Scholar
  • 21 van der Lei J. Use and abuse of computer-stored medical records. Methods Inf Med 1991; Apr; 30 (02) 79-80.
    PubMedGoogle Scholar
  • 22 Nolan J, McNair P, Brender J. Factors influencing transferability of knowledge-based systems. Int J Biomed Comput 1991; 27: 7-26.
    PubMedGoogle Scholar
  • 23 Garza M, Del Fiol G, Tenenbaum J, Walden A, Zozus MN. Evaluating common data models for use with a longitudinal community registry. J Biomed Inform 2016; Dec; 64: 333-41.
    PubMedGoogle Scholar
  • 24 Trifiro G, Fourrier-Reglat A, Sturkenboom MC, Diaz CAcedo, van der Lei J. The EU-ADR project: preliminary results and perspective. Stud Health Technol Inform 2009; 148: 43-9.
    PubMedGoogle Scholar
  • 25 Brender J, McNair P. Meta-level concepts as a basis for structuring data. In: Hejlesen O, Bygholm A, Bertelsen P. editors. 4th Scandinavian Conference on Health Informatics. Aalborg, Denmark: Virtual Centre for Health Informatics, Aalborg University; 2006: 17-20.
    Google Scholar
  • 26 Brender J, McNair P. Enhancing semantic interoperability: a model for making context explicit. In: Harorimana D, Watkins D. editors. 9th European Conference on Knowledge Management. Southampton Solent University; UK: 2008: 83-91.
    Google Scholar
  • 27 Scott PJ, Cornet R, McCowan C, Peek N, Fraccaro P, Geifman N. et al. Informatics for Health 2017: Advancing both science and practice. J Innov Health Inform 2017; 24 (01) 1-2.
    PubMedGoogle Scholar
  • 28 Longhurst CA, Harrington RA, Shah NH. A ‘green button’ for using aggregate patient data at the point of care Health Aff (Millwood). 2014; 33 (07) 1229-35.
    PubMedGoogle Scholar
  • 29 Frankovich J, Longhurst CA, Sutherland SM. Evidence-based medicine in the EMR era. N Engl J Med 2011; 365 (19) 1758-9.
    PubMedGoogle Scholar
  • 30 Gabriel SE, Normand SL. Getting the methods right--the foundation of patient-centered outcomes research. N Engl J Med 2012; 367 (09) 787-90.
    CrossrefPubMedGoogle Scholar
  • 31 Tunis SR, Clarke M, Gorst SL, Gargon E, Blazeby JM, Altman DG. et al. Improving the relevance and consistency of outcomes in comparative effectiveness research. J Comp Eff Res 2016; 05 (02) 193-205.
    PubMedGoogle Scholar
  • 32 Collins FS, Varmus H. A new initiative on precision medicine. N Engl J Med 2015; 372 (09) 793-5.
    CrossrefPubMedGoogle Scholar
  • 33 Coleman J. Segmenting data privacy. Cross-industry initiative aims to piece out privacy within the health record. J AHIMA 2013; 84 (02) 34-8.
    PubMedGoogle Scholar
  • 34 Clancy CM. Common formats allow uniform collection and reporting of patient safety data by patient safety organizations. Am J Med Qual 2010; 25 (01) 73-5.
    PubMedGoogle Scholar
  • 35 Armstrong S. The computer will assess you now. BMJ 2016; 355: i5680.
    PubMedGoogle Scholar
  • 36 OECD. Improving Health Sector Efficiency: The Role of Information and Communication Technologies Paris: OECD Publishing. 2010 [cited 2016 21 November]. Available from: http://www.oecd-ilibrary.org/social-issues-migration-health/improving-health-sector-efficiency_9789264084612-en
    PubMedGoogle Scholar
  • 37 Bélanger F, Crossler RE. Privacy in the digital age: a review of information privacy research in information systems. MIS Quarterly 2011; 35 (04) 1017-42.
    PubMedGoogle Scholar
  • 38 Pavlou PA. State of the information privacy literature: where are we now and where should we go?. MIS Quarterly 2011; 35 (04) 977-88.
    PubMedGoogle Scholar
  • 39 Ruotsalainen PS, Blobel BG, Seppala AV, Sorvari HO, Nykanen PA. A conceptual framework and principles for trusted pervasive health. J Med Internet Res 2012; 14 (02) e52.
    PubMedGoogle Scholar
  • 40 European Commission. eHealth Action Plan 2012-2020 - Innovative healthcare for the 21st century 2012 [cited 2016 21 November]. Available from: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52012DC0736&from=EN
    PubMedGoogle Scholar
  • 41 OECD. Draft OECD guide to measuring ICTs in the Health Sector 2015 [cited 2016 21 November]. Available from: https://www.oecd.org/health/health-systems/Draft-oecd-guide-to-measuringicts-in-the-health-sector.pdf
    PubMedGoogle Scholar
  • 42 Hyppönen H, Kangas M, Reponen J, Nohr C, Villumsen S, Koch S, et al. Nordic eHealth benchmarking. Status 2014. 2014 [cited 2016 21 November]. Available from: https://norden.diva-portal.org/smash/get/diva2:821230/FULLTEXT01.pdf
    PubMedGoogle Scholar
  • 43 Gude WT, van der Veer SN, de Keizer NF, Coiera E, Peek N. Optimizing Digital Health Informatics Interventions Through Unobtrusive Quantitative Process Evaluations. Stud Health Technol Inform 2016; 228: 594-8.
    PubMedGoogle Scholar
  • 44 Dentler K, Numans ME, ten Teije A, Cornet R, de Keizer NF. Formalization and computation of quality measures based on electronic medical records. J Am Med Inform Assoc 2014; Mar-Apr; 21 (02) 285-91.
    PubMedGoogle Scholar
  • 45 Dentler K, Cornet R, Ten Teije A, Tytgat K, Klinkenbujl J, De Keizer N. The reproducibility of CLIF, a method for clinical quality indicator formalisation. Stud Health Technol Inform 2012; 180: 113-7.
    PubMedGoogle Scholar
  • 46 Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD. et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012 Jun 13(6) CD000259.
    PubMedGoogle Scholar
  • 47 Roshanov PS, Misra S, Gerstein HC, Garg AX, Sebaldt RJ, Mackay JA. et al. Computerized clinical decision support systems for chronic disease management: a decision-maker-researcher partnership systematic review. Implement Sci 2011; Aug 03; 06: 92.
    PubMedGoogle Scholar
  • 48 Musen MA, Middleton B, Greenes RA. Clinical Decision-Support Systems. In: Shortliffe EH, Cimino JJ. editors. Biomedical Informatics: Computer Applications in Health Care and Biomedicine. London: Springer; 2014: 643-74.
    Google Scholar
  • 49 Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ. et al. No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci 2014; Jan 17; 09: 14.
    PubMedGoogle Scholar
  • 50 Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French SD. et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med 2014; Nov; 29 (11) 1534-41.
    PubMedGoogle Scholar
  • 51 Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 2013; Jan 12; 14: 15.
    PubMedGoogle Scholar
  • 52 Davidson EJ. Evaluation methodology basics: The nuts and bolts of sound evaluation. London: Sage; 2005
    Google Scholar
  • 53 Bureau of Health Information. Data Matters -Linking data to unlock information. The use of linked data in healthcare performance assessment. Sydney, NSW: Bureau of Health Information; 2015 [cited 2016 21 November]. Available from: http://www.bhi.nsw.gov.au/__data/assets/pdf_file/0003/290712/DataMatters.pdf
    PubMedGoogle Scholar
  • 54 Honeyman M, Dunn P, McKenna H. A digital NHS? An introduction to the digital agenda and plans for implementation. London: The King’s Fund; 2016 [cited 2016 21 November]. Available from: https://www.kingsfund.org.uk/publications/digital-nhs
    PubMedGoogle Scholar
  • 55 National Patient Safety Foundation. Free from Harm - Accelerating Patient Safety Improvement Fifteen Years after To Err Is Human. Boston, MA: National Patient Safety Foundation; 2015 [cited 2016 21 November]. Available from: https://psnet.ahrq.gov/resources/resource/29554/free-from-harm-accelerating-patient-safety-improvement-fifteen-years-after-to-err-is-human
    PubMedGoogle Scholar
  • 56 Singh H, Sittig DF. Measuring and improving patient safety through health information technology: The Health IT Safety Framework. BMJ Qual Saf 2016; Apr; 25 (04) 226-32.
    CrossrefPubMedGoogle Scholar
  • 57 Okun S, McGraw D, Stang PEL, Goldmann D, Kupersmith J. et al. Making the case for continuous learning from routinely collected data. Washington, DC: Institute of Medicine; 2013 [cited 2016 21 November]. Available from: https://nam.edu/wp-content/uploads/2015/06/MakingtheCase.pdf
    PubMedGoogle Scholar
  • 58 Raghupathi W, Raghupathi V. Big data analytics in healthcare: promise and potential. Health Inf Sci Syst 2014; 02: 3.
    PubMedGoogle Scholar
  • 59 Shaw J, Taylor R, Dix K. Uses and abuses of performance data in healthcare. London: Dr Foster; 2015 [cited 2016 21 November]. Available from: http://www.drfoster.com/updates/recent-publica-tions/uses-and-abuses-of-performance-data-in-healthcare/
    PubMedGoogle Scholar
  • 60 Abdelhak M, Grostick S, Hankin M, Jacobs E. Health Information: Management of a Strategic Resource. Philadelphia: WB Saunders; 1996
    Google Scholar
  • 61 Streiner DL, Norman GR, Cairney J. Health Measurement Scales. A practical guide to their development and use. 5th ed. Oxford: OUP; 2014
    Google Scholar
  • 62 Strome TL. Healthcare analytics for quality and performance improvement. Chichester: John Wiley & Sons; 2013
    Google Scholar
  • 63 Brender J, Talmon J, de Keizer N, Nykanen P, Rigby M, Ammenwerth E. STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration. Appl Clin Inform 2013; 04 (03) 331-58.
    PubMedGoogle Scholar
  • 64 McNair JB. Handbook of evaluation methods for health informatics. London: Elsevier Academic Press; 2006
    Google Scholar
  • 65 Nykanen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC. et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform 2011; Dec; 80 (12) 815-27.
    PubMedGoogle Scholar
  • 66 Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykanen P, Rigby M. STARE-HI--Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009; Jan; 78 (01) 1-9.
    CrossrefPubMedGoogle Scholar
  • 67 Hackl WO, Ammenwerth E. SPIRIT: Systematic Planning of Intelligent Reuse of Integrated Clinical Routine Data. A Conceptual Best-practice Framework and Procedure Model. Methods Inf Med 2016; 55 (02) 114-24.
    PubMedGoogle Scholar
  • 68 Fayyad U, Piatetsky-Shapiro G, Smyth P. From data mining to knowledge discovery in databases. AI Magazine 1996; 17 (03) 37.
    PubMedGoogle Scholar
  • 69 Codd E, Codd S, Salley C. Providing OLAP (On-line analytical processing) to user-analysts: an IT mandate. Ann Arbor, Michigan: Codd & Associates; 1993
    Google Scholar
  • 70 Ammenwerth E, Rigby M. editors. Evidence-based Health Informatics: Promoting safety and efficiency through scientific methods and ethical policy. Amsterdam: IOS Press; 2016
    Google Scholar