PDF icon Download PDF
Open Access
CC BY-NC-ND 4.0 · Methods Inf Med 2022; 61(S 01): e35-e44
DOI: 10.1055/s-0042-1743170
Original Article

Evaluating Prediction of Continuous Clinical Values: A Glucose Case Study

George Hripcsak
1   Department of Biomedical Informatics, Columbia University, New York, New York, United States
2   Medical Informatics Services, NewYork-Presbyterian Hospital, New York, New York, United States
,
David J. Albers
1   Department of Biomedical Informatics, Columbia University, New York, New York, United States
3   Department of Pediatrics, University of Colorado Denver—Anschutz Medical Campus, Denver, Colorado, United States
› Author Affiliations
Funding This work was funded by grants from the National Institutes of Health R01 LM006910 “Discovering and applying knowledge in clinical databases” and R01 LM012734 “Mechanistic machine learning.”
› Further Information
  Permissions and Reprints
Preview

Abstract

Background It would be useful to be able to assess the utility of predictive models of continuous values before clinical trials are performed.

Objective The aim of the study is to compare metrics to assess the potential clinical utility of models that produce continuous value forecasts.

Methods We ran a set of data assimilation forecast algorithms on time series of glucose measurements from neurological intensive care unit patients. We evaluated the forecasts using four sets of metrics: glucose root mean square (RMS) error, a set of metrics on a transformed glucose value, the estimated effect on clinical care based on an insulin guideline, and a glucose measurement error grid (Parkes grid). We assessed correlation among the metrics and created a set of factor models.

Results The metrics generally correlated with each other, but those that estimated the effect on clinical care correlated with others the least and were generally associated with their own independent factors. The other metrics appeared to separate into those that emphasized errors in low glucose versus errors in high glucose. The Parkes grid was well correlated with the transformed glucose but not the estimation of clinical care.

Discussion Our results indicate that we need to be careful before we assume that commonly used metrics like RMS error in raw glucose or even metrics like the Parkes grid that are designed to measure importance of differences will correlate well with actual effect on clinical care processes. A combination of metrics appeared to explain the most variance between cases. As prediction algorithms move into practice, it will be important to measure actual effects.

Keywords

metrics - predictive models - data assimilation - machine learning

Authors' Contributions

All authors made substantial contributions to the conception and design of the work; drafted the work or revised it critically for important intellectual content; had final approval of the version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.




Publication History

Received: 18 June 2021

Accepted: 28 December 2021

Article published online:
23 February 2022

© 2022. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany