Automated versus Manual Data Extraction of the Padua Prediction Score for Venous Thromboembolism Risk in Hospitalized Older Adults

Abstract

Objective Venous thromboembolism (VTE) prophylaxis is an important consideration for hospitalized older adults, and the Padua Prediction Score (PPS) is a risk prediction tool used to prioritize patient selection. We developed an automated PPS (APPS) algorithm using electronic health record (EHR) data. This study examines the accuracy of APPS and its individual components versus manual data extraction.

Methods This is a retrospective cohort study of hospitalized general internal medicine patients, aged 70 and over. Fourteen clinical variables were collected to determine their PPS; APPS used EHR data exports from health system databases, and a trained abstractor performed manual chart abstractions. We calculated sensitivity and specificity of the APPS, using manual PPS as the gold standard for classifying risk category (low vs. high). We also examined performance characteristics of the APPS for individual variables.

Results PPS was calculated by both methods on 311 individuals. The mean PPS was 3.6 (standard deviation, 1.8) for manual abstraction and 2.8 (1.4) for APPS. In detecting patients at high risk for VTE, the sensitivity and specificity of the APPS algorithm were 46 and 94%, respectively. The sensitivity for APPS was poor (range: 6–34%) for detecting acute conditions (i.e., acute myocardial infarction), moderate (range: 52–74%) for chronic conditions (i.e., heart failure), and excellent (range: 94–98%) for conditions of obesity and restricted mobility. Specificity of the automated extraction method for each PPS variable was > 87%.

Conclusion APPS as a stand-alone tool was suboptimal for classifying risk of VTE occurrence. The APPS accurately identified high risk patients (true positives), but lower scores were considered indeterminate.

Keywords

decision support algorithm - venous thromboembolism - geriatrics - electronic health record and systems - inpatient

Protection of Human and Animal Subjects

This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was approved by the Duke University Institutional Review Board.

• References

• 1 Cohen AT, Agnelli G, Anderson FA. , et al; VTE Impact Assessment Group in Europe (VITAE). Venous thromboembolism (VTE) in Europe. The number of VTE events and associated morbidity and mortality. Thromb Haemost 2007; 98 (04) 756-764
• 2 Kahn SR, Hirsch A, Shrier I. Effect of postthrombotic syndrome on health-related quality of life after deep venous thrombosis. Arch Intern Med 2002; 162 (10) 1144-1148
• 3 Pengo V, Lensing AW, Prins MH. , et al; Thromboembolic Pulmonary Hypertension Study Group. Incidence of chronic thromboembolic pulmonary hypertension after pulmonary embolism. N Engl J Med 2004; 350 (22) 2257-2264
• 4 Anderson Jr FA, Zayaruzny M, Heit JA, Fidan D, Cohen AT. Estimated annual numbers of US acute-care hospital patients at risk for venous thromboembolism. Am J Hematol 2007; 82 (09) 777-782
• 5 Cohen AT, Tapson VF, Bergmann JF. , et al; ENDORSE Investigators. Venous thromboembolism risk and prophylaxis in the acute hospital care setting (ENDORSE study): a multinational cross-sectional study. Lancet 2008; 371 (9610): 387-394
• 6 Gould MK, Garcia DA, Wren SM. , et al. Prevention of VTE in nonorthopedic surgical patients: Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest 2012; 141 (2, Suppl): e227S-e277S
• 7 Qaseem A, Chou R, Humphrey LL, Starkey M, Shekelle P. ; Clinical Guidelines Committee of the American College of Physicians. Venous thromboembolism prophylaxis in hospitalized patients: a clinical practice guideline from the American College of Physicians. Ann Intern Med 2011; 155 (09) 625-632
• 8 Barbar S, Noventa F, Rossetto V. , et al. A risk assessment model for the identification of hospitalized medical patients at risk for venous thromboembolism: the Padua Prediction Score. J Thromb Haemost 2010; 8 (11) 2450-2457
• 9 Marcucci M, Iorio A, Nobili A. , et al; REPOSI (REgistro POliterapie Società Italiana di Medicina Interna) Investigators. Prophylaxis of venous thromboembolism in elderly patients with multimorbidity. Intern Emerg Med 2013; 8 (06) 509-520
• 10 Depietri L, Marietta M, Scarlini S. , et al. Clinical impact of application of risk assessment models (Padua Prediction Score and Improve Bleeding Score) on venous thromboembolism, major hemorrhage and health expenditure associated with pharmacologic VTE prophylaxis: a “real life” prospective and retrospective observational study on patients hospitalized in a Single Internal Medicine Unit (the STIME study). Intern Emerg Med 2018; 13 (04) 527-534
• 11 Pavon JM, Sloane RJ, Pieper CF. , et al. Poor adherence to risk stratification guidelines results in overuse of venous thromboembolism prophylaxis in hospitalized older adults. J Hosp Med 2018; 13 (06) 403-404
• 12 Mamykina L, Vawdrey DK, Stetson PD, Zheng K, Hripcsak G. Clinical documentation: composition or synthesis?. J Am Med Inform Assoc 2012; 19 (06) 1025-1031
• 13 Hripcsak G, Vawdrey DK, Fred MR, Bostwick SB. Use of electronic clinical documentation: time spent and team interactions. J Am Med Inform Assoc 2011; 18 (02) 112-117
• 14 Gershon AS, Wang C, Guan J, Vasilevska-Ristovska J, Cicutto L, To T. Identifying individuals with physician diagnosed COPD in health administrative databases. COPD 2009; 6 (05) 388-394
• 15 Alotaibi GS, Wu C, Senthilselvan A, McMurtry MS. The validity of ICD codes coupled with imaging procedure codes for identifying acute venous thromboembolism using administrative data. Vasc Med 2015; 20 (04) 364-368
• 16 Jones BE, Jones J, Bewick T. , et al. CURB-65 pneumonia severity assessment adapted for electronic decision support. Chest 2011; 140 (01) 156-163
• 17 Vinson DR, Morley JE, Huang J. , et al; Kaisers Permanente CREST Network. The accuracy of an electronic pulmonary embolism severity index auto-populated from the electronic health record: setting the stage for computerized clinical decision support. Appl Clin Inform 2015; 6 (02) 318-333
• 18 Benesch C, Witter Jr DM, Wilder AL, Duncan PW, Samsa GP, Matchar DB. Inaccuracy of the International Classification of Diseases (ICD-9-CM) in identifying the diagnosis of ischemic cerebrovascular disease. Neurology 1997; 49 (03) 660-664
• 19 Quan H, Li B, Saunders LD. , et al; IMECCHI Investigators. Assessing validity of ICD-9-CM and ICD-10 administrative data in recording clinical conditions in a unique dually coded database. Health Serv Res 2008; 43 (04) 1424-1441
• 20 Elias P, Khanna R, Dudley A. , et al. Automating venous thromboembolism risk calculation using electronic health record data upon hospital admission: the automated Padua Prediction Score. J Hosp Med 2017; 12 (04) 231-237
• 21 Navar-Boggan AM, Rymer JA, Piccini JP. , et al. Accuracy and validation of an automated electronic algorithm to identify patients with atrial fibrillation at risk for stroke. Am Heart J 2015; 169 (01) 39-44
• 22 McCormick N, Lacaille D, Bhole V, Avina-Zubieta JA. Validity of heart failure diagnoses in administrative databases: a systematic review and meta-analysis. PLoS One 2014; 9 (08) e104519
• 23 Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform 2013; 46 (05) 830-836
• 24 Barber C, Lacaille D, Fortin PR. Systematic review of validation studies of the use of administrative data to identify serious infections. Arthritis Care Res (Hoboken) 2013; 65 (08) 1343-1357
• 25 Wang LE, Shaw PA, Mathelier HM, Kimmel SE, French B. Evaluating risk-prediction models using data from electronic health records. Ann Appl Stat 2016; 10 (01) 286-304
• 26 Greene MT, Spyropoulos AC, Chopra V. , et al. Validation of risk assessment models of venous thromboembolism in hospitalized medical patients. Am J Med 2016; 129 (09) 1001.e9-1001.e18
• 27 Deleger L, Brodzinski H, Zhai H. , et al. Developing and evaluating an automated appendicitis risk stratification algorithm for pediatric patients in the emergency department. J Am Med Inform Assoc 2013; 20 (e2): e212-e220
• 28 Ford E, Carroll JA, Smith HE, Scott D, Cassell JA. Extracting information from the text of electronic medical records to improve case detection: a systematic review. J Am Med Inform Assoc 2016; 23 (05) 1007-1015
• 29 Bates DW, Kuperman GJ, Wang S. , et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530