Sensor, Signal, and Imaging Informatics

Summary

Objective: To summarize significant contributions to sensor, signal, and imaging informatics published in 2016.

Methods: We conducted an extensive search using PubMed® and Web of Science® to identify the scientific contributions published in 2016 that addressed sensors, signals, and imaging in medical informatics. The three section editors selected 15 candidate best papers by consensus. Each candidate article was reviewed by the section editors and at least two other external reviewers. The final selection of the six best papers was conducted by the editorial board of the Yearbook.

Results: The selected papers of 2016 demonstrate the important scientific advances in management and analysis of sensor, signal, and imaging information.

Conclusion: The growing volume of signal and imaging data provides exciting new challenges and opportunities for research in medical informatics. Evolving technologies provide faster and more effective approaches for pattern recognition and diagnostic evaluation. The papers selected here offer a small glimpse of the high-quality scientific work published in 2016 in the domain of sensor, signal, and imaging informatics.

Keywords

Signal processing - biomedical imaging - deep learning - computer aided diagnosis - data integration

Introduction

The field of “Sensor, Signal, and Imaging Informatics” (SSII) offers an active and growing domain in biomedical informatics. In addition to systems to gather physiological data on hospital patients and dedicated signal recording devices for outpatient use, the variety and quality of signal and sensor data are growing rapidly through the increased adoption of wearable devices. Imaging informatics, too, remains a dynamic area of research and clinical applications as the number of clinical departments that utilize imaging-derived information continues to grow, including radiology, pathology, dermatology, cardiology, ophthalmology, and other image-rich medical specialties. Although 2016 has brought numerous advances in machine learning and the rise of deep learning techniques in a wide spectrum of clinical applications, no field has felt the impact of these developments more than the fields related to signals and imaging. Given that signal and imaging datasets tend to be high-dimensional and longitudinal in nature, these large datasets are ripe for machine learning approaches such as deep learning to identify spatial and temporal patterns.

The large quantities of data generated in these fields have provided ample material for those interested in “big data” and for machine learning systems that seek to interpret findings, make diagnoses, and predict clinical outcomes. For example, a recent example of training a deep learning to detect diabetic retinopathy on a large cohort of over 120,000 retinal images achieved up to 97% accuracy [[1]]. Such applications of machine learning are also starting to be published in mainstream clinical journals such as the Journal of the American Medical Association. The rise of interest in this area has led individuals to wonder aloud whether machine learning will start displacing jobs of clinicians such as radiologists and pathologists [[2]]. Although there remain numerous challenges related to data quality, interoperability, integration, and model validation and deployment, medical advances will rely on the effective use of machine learning to extract actionable information from the growing amounts of “big data” being generated.

Our selection of publications from 2016 highlights the significant contributions that medical informatics research has had in the signals and imaging communities, addressing challenges related to data quality, machine learning, clinical decision support, and workflow. The six selected papers offer a glimpse of the quality and breadth of the work being conducted in SSII. In addition, the survey paper by Chennubhotla and colleagues reviews the developments in a specific area, cancer imaging, that embodies the aspirations and opportunities that the SSII community is facing [[3]].

About the Paper Selection

To compile the list of eligible papers, we performed a search of two electronic databases, PubMed/Medline^® and Web of Science^®. The search was conducted in December 2016 to identify peer-reviewed journal articles published in 2016, in the English language, related to SSII research in medical informatics. As in previous years, a wide spectrum of MeSH^® keywords and topics was considered, including image and signal processing, pattern recognition and information extraction, telemedicine, sensor monitoring, and computer-aided diagnosis. We used free-text (tagged as “tw” in the bibliographic record) and MeSH^® terms (tagged as “mh”) as keywords. PubMed/Medline^® was queried to test keywords in an iterative process. Consequently, two queries were built: one based on MeSH^® terms used to search the major topics in the PubMed/Medline^® database, the second one based on free-text keywords searched in title or abstracts through PubMed/Medline^® and Web of Science^® databases. We also added keywords that have been trending this past year such as deep learning. One of the three section editors (WH) performed the searches. In addition to the search of electronic databases, manual searches of key themes were performed in leading journals of biomedical informatics, imaging informatics, and imaging specialty journals, such as Journal of the American Medical Informatics Association, Journal of Digital Imaging, and Radiology.

The search results were reconciled into a single list of 686 papers. The three section editors independently screened the titles and abstracts to identify relevant papers. The section editors classified the papers into three categories: accepted, rejected, or pending. They then reviewed in detail the accepted and pending full-text articles to finally reach a consensual list of 15 candidate papers. Papers were considered according to their originality, scientific and/or clinical impact, and scientific quality. Individual editors recused themselves in decisions related to papers that posed potential for conflict of interest. In accordance with the IMIA Yearbook selection process, the 15 candidate best papers were evaluated by the section editors and by at least two additional external reviewers. Six papers were selected as best papers ([Table 1]). A content summary of the selected best papers can be found in the appendix of this synopsis.

Table 1

Best paper selection of articles for the IMIA Yearbook of Medical Informatics 2017 in the section ‘Sensor, Signal and Imaging Informatics’. The articles are listed in alphabetical order of the first author’s surname.

Section
Sensor, Signal and Imaging Informatics
Arnold CW, Wallace WD, Chen S, Oh A, Abtin F, Genshaft S, Binder S, Aberle D, Enzmann D. RadPath: A web-based system for integrating and correlating radiology and pathology findings during cancer diagnosis. Acad Radiol 2016 Jan;23(1):90-100. Hravnak M, Chen L, Dubrawski A, Bose E, Clermont G, Pinsky MR. Real alerts and artifact classification in archived multi-signal vital sign monitoring data: implications for mining big data. J Clin Monit Comput 2016 Dec;30(6):875-88. Kalpathy-Cramer J, Zhao B, Goldgof D, Gu Y, Wang X, Yang H, Tan Y, Gillies R, Napel S. A comparison of lung nodule segmentation algorithms: methods and results from a multi-institutional study. J Digit Imaging 2016 Aug;29(4):476-87. Moss TJ, Lake DE, Calland JF, Enfield KB, Delos JB, Fairchild KD, Moorman JR. Signatures of subacute potentially catastrophic illness in the ICU: model development and validation. Crit Care Med 2016 Sep;44(9):1639-48. Petousis P, Han SX, Aberle D, Bui AA. Prediction of lung cancer incidence on the low-dose computed tomography arm of the National Lung Screening Trial: A dynamic Bayesian network. Artif Intell Med 2016 Sep;72:42-55. Springer DB, Tarassenko L, Clifford GD. Logistic regression-HSMM-based heart sound segmentation. IEEE Trans Biomed Eng 2016 Apr;63(4):822-32.

Conclusions and Outlook

The use of pattern recognition techniques to provide increasingly intelligent diagnostic understanding is a key area of work in sensor, signal, and imaging informatics. The candidate papers offered innovative approaches to address the evolving challenges of analyzing sensor, signal, and imaging data. In signal and sensor informatics, the data typically represent multiple single-dimensional values that change over time; the input data’s multidimensionality and temporal fluctuations pose significant challenges. Large sets of signal data can allow detection of subtle patterns of physiologic changes in critically ill patients; such changes may precede clinically overt deterioration, and may allow treatment to be more effective. Moss et al. analyzed 146 patient-years of vital-sign and ECG time-series data on 9,232 ICU patients to identify physiologic signatures in the hours before clinical detection and intervention [[4]]. Springer and colleagues addressed the accurate segmentation of the first and second heart sound within noisy real-world phonocardiogram recordings using a hidden semi-Markov model (HSMM) extended with the use of logistic regression for emission probability estimation [[5]]. Their analysis of 10,172 seconds of recordings from 112 patients achieved an average F1 score of 95.63 ± 0.85%.

In an analysis of physiologic markers of acute emotional stress in young healthy volunteers, Hernando et al. reported that analysis of respiration and heart-rate variability together more reliably characterized autonomic nervous response [[6]]. Artifacts pose a significant challenge to accurate classification of signal data. Hravnak and colleagues developed an approach to distinguish real alerts from artifacts in archived multi-signal vital sign monitoring data [[7]]. Accurate “data cleaning” steps are critical for mining “big data” sets of high-volume, real-time physiologic data.

In imaging informatics, input data typically represent two- or three-dimensional spatial information; some studies may include “dynamic” imaging to visualize motion of the heart or a joint, or to view physiological changes that vary with time and/or external stimulation. For example, changes in brain structures over time – particularly of the hippocampus – may offer an important bio-marker of Alzheimer disease. Chincarini et al. found that longitudinal volume measurements of the hippocampus can provide meaningful clinical insight regarding the development and progression of Alzheimer’s disease [[8]]. Temporal information also plays a role in imaging-based decision making. Petousis et al. applied a dynamic Bayesian network (DBN), a model that repeats the static interactions of a conventional Bayesian network over time, to data from the National Lung Screening Trial to improve the prediction of lung cancer using longitudinal data [[9]].

The papers in the SSII domain addressed novel approaches to improve clinical use of diagnostic information for cross-specialty correlation and follow-up. Arnold et al. described a web-based platform called RadPath to correlate and integrate findings from diagnostic radiology and pathology [[10]]. This work engenders and facilitates communication between specialties to address the increasing complexity and team-based approaches of modern medical care. Tracking the recommendations from diagnostic examinations is critical, too. Many actionable findings – such as an incidentally detected kidney lesion on a CT performed for trauma – demand follow-up care that may not be addressed during the patient‘s acute presentation [[11]]. Such systems seek to “close the loop” to assure that recommended follow-up examinations are completed.

Deep learning algorithms require a large amount of data to fine-tune their parameters, and acquiring sufficiently large sets of “labeled” data may not be feasible. Sun et al. combined a relatively small amount of labeled data with a large amount of unlabeled data to enhance a deep convolutional neural network model for breast cancer diagnosis [[12]]. Their approach significantly improved the performance of their diagnostic model. Summers provides an excellent review of machine learning approaches in abdominal CT interpretation [[13]]. This work, published in a leading clinical radiology journal, points up the progress made in fully automated identification of organs and blood vessels in abdominal CT images, and the impressive performance of deep learning-based methods in many of the described applications.

Emerging research also is looking at how to effectively manage and integrate multi-modal datasets. Peikari and colleagues used a texture-based approach to “triage” diagnostically relevant regions from digitized pathology whole slides of breast cancer [[14]]. Their approach outperformed two previously proposed methods and achieved an AUC of 0.87 on 5,151 specimens from 65 tissue slides. Milchenko et al. developed a framework to pre-process multi-spectral MRI data acquired from heterogeneous imaging systems [[15]]. They applied this framework in the domain of neuro-oncology MRI data acquired from various institutions, different MRI devices, and varying sequence parameters. The ability to capture whole-embryo morphology and gene expression has made three-dimensional (3D) imaging an important modality to understand gene-phenotype relationships in developmental biology. Brown and colleagues presented a software suite to facilitate uploading, analysis, and dissemination of 3D images of embryos [[16]].

Competitions and open databases provide materials that allow investigators to test new algorithms and compare their performance to other approaches. Comparative analyses of algorithms in the literature have been limited by the lack of high-quality, standardized, and rigorously validated data. Liu and colleagues developed an open-access database for the evaluation of automated heart sound segmentation and classification [[17]]. Kalpathy-Cramer et al. reported a multi-institutional effort to assess the repeatability and reproducibility of nodule borders and volume estimate bias of computerized segmentation algorithms in CT images of lung cancer [[18]].

In conclusion, the selected papers are a small sampling of the many efforts that are advancing the science of machine learning in healthcare. In connection with the 2016 Yearbook theme of “Learning from experience: secondary use of patient data”, many of the developments described in the cited papers would not be possible without the availability of large, curated patient datasets from which algorithms can be trained and validated. As new datasets that combine clinical, imaging, molecular, and signal data become available, new insights can be discovered, leading to more powerful and robust tools for pattern recognition.

Appendix: Content Summaries of the Selected Best Papers for the IMIA Yearbook 2017 in the “Sensor, Signal, and Imaging Informatics” Section

Arnold CW, Wallace WD, Chen S, Oh A, Abtin F, Genshaft S, Binder S, Aberle D, Enzmann D

RadPath: A web-based system for integrating and correlating radiology and pathology findings during cancer diagnosis

Acad Radiol 2016 Jan;23(1):90–100

Cancer diagnosis should involve close coordination and communication among physicians in radiology, pathology, and clinical care in order to assure timely, efficient, and medically correct treatment decisions. The authors developed the web-based RadPath system to correlate and integrate diagnostic radiology and pathology report findings. RadPath retrieves reports, images, and test results from the institution’s clinical information system to construct a database of relevant information. Radiologists can correlate imaging findings with the pathology diagnosis; they otherwise might not receive feedback about pathology results related to cases in which they have interpreted diagnostic studies or performed biopsies. Medicine’s increasing complexity and team-based management approaches have demanded improved clinical communication and information exchange. The authors’ work is a welcome addition.

Hravnak M, Chen L, Dubrawski A, Bose E, Clermont G, Pinsky MR

Real alerts and artifact classification in archived multi-signal vital sign monitoring data: implications for mining big data

J Clin Monit Comput 2016 Dec;30(6):875-88

Hospital inpatients at high risk for cardio-respiratory instability undergo continuous non-invasive monitoring of their vital signs, such as heart rate, respiratory rate, blood pressure, and peripheral arterial oxygen saturation. To assure accurate diagnosis, it is critical to distinguish truly aberrant vital sign measurements from false measurements caused by artifacts such as patient movement, electrical noise, or loose sensors. The authors used machine-learning algorithms trained on labeled vital sign data to automatically classify alerts as real or artifact. Their algorithms achieved an area under the curve (AUC) of 0.72 to 0.94 on a test dataset, which suggests that their approach could be used for accurate automated classification of vital sign alerts as real or artifact. Such information provides a “data cleaning” step that can support data mining for future model building.

Kalpathy-Cramer J, Zhao B, Goldgof D, Gu Y, Wang X, Yang H, Tan Y, Gillies R, Napel S

A comparison of lung nodule segmentation algorithms: methods and results from a multi-institutional study

J Digit Imaging 2016 Aug;29(4):476-87

Automated systems play an increasingly important role in identifying, classifying, and measuring lesions in medical images. These systems must provide accurate and reproducible volumetric measurements for diagnosis, staging, and assessment of treatment response. For lung nodules on CT, it is often the case that images are acquired on different scanners, generated with different imaging parameters (e.g., slice thickness, reconstruction kernel), and analyzed by different nodule-measurement systems. The authors evaluated three algorithms on 52 nodules from The Cancer Imaging Archive acquired from 40 actual patients and from an anatomic phantom. The authors found considerable differences between the algorithms’ measurements, especially in a subset of heterogeneous nodules. They recommend that longitudinal studies should use the same software at all time points.

Moss TJ, Lake DE, Calland JF, Enfield KB, Delos JB, Fairchild KD, Moorman JR

Signatures of subacute potentially catastrophic illness in the ICU: model development and validation

Crit Care Med 2016 Sep;44(9):1639-48

Critically ill patients may develop life-threatening conditions such as sepsis, respiratory failure, or hemorrhage, which present frequently with abnormal vital signs. To detect and treat such conditions most effectively, one should identify these conditions’ more subtle physiologic changes. The investigators analyzed 146 patient-years of vital signs and ECG data on ICU patients to identify early physiologic signatures of cardiorespiratory failure. They found that they could detect potentially catastrophic, subacute illnesses in ICU patients hours before clinical detection and intervention. By detecting potentially catastrophic conditions more rapidly, they hope to draw attention to the patients at highest risk, enable earlier interventions, and attain better patient-care outcomes.

Petousis P, Han SX, Aberle D, Bui AA

Prediction of lung cancer incidence on the low-dose computed tomography arm of the National Lung Screening Trial: A dynamic Bayesian network

Artif Intell Med 2016 Sep;72:42–55

The most effective way to improve cancer survival in high-risk individuals is to identify disease at an early stage. The National Lung Screening Trial (NLST) showed that low-dose computed tomography (LDCT) reduced mortality more effectively than radiography in lung cancer screening, and has led to the implementation of CT-based lung cancer screening programs. However, LDCTs result in a large number of false-positive results. The authors explored dynamic Bayesian networks (DBN) to help understand how longitudinal data might inform lung cancer screening decisions. The lung cancer screening DBNs demonstrated high discrimination and predictive power with the majority of cancer and non-cancer cases, and the DBN models outperformed logistic regression and naïve Bayes approaches. This work highlights the use of a large national dataset to create a model for decision support and demonstrates the potential of a prediction model to outperform radiologist interpretation in specific instances.

Springer DB, Tarassenko L, Clifford GD

Logistic regression-HSMM-based heart sound segmentation

IEEE Trans Biomed Eng 2016 Apr;63(4):822-32

Segmentation of heart sounds may be relatively simple in noise-free recordings, but noise – such as speech, patient motion, and intestinal or breathing sounds – makes the task more difficult. Neural networks and other static machine-learning approaches that lack temporal duration and ordering information may not perform as well in differentiating heart sound-like noises from noisy heart sounds. The authors sought to improve identification of first and second heart sounds in phonocardiogram (PCG) data. Their model incorporated a priori knowledge about the expected duration of the heart sounds encoded in a hidden semi-Markov model (HSMM). The authors evaluated their method on a recorded PCG dataset of more than 10,000 seconds from 112 patients. Their system’s average F1 score of 96% exceeded that of 86% achieved by conventional systems. Probabilistic models, such as hidden Markov models, can improve performance over threshold-based segmentation methods in the analysis of heart sound recordings.

Acknowledgement

We thank Brigitte Séroussi, Adrien Ugon, and John Holmes for editorial guidance and support. We extend our appreciation to the reviewers for participating in the selection process.

• References

• 1 Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 2016; 316 (22) 2402-10.
• 2 Obermeyer Z, Emanuel EJ. Predicting the future — big data, machine learning, and clinical medicine. New Engl J Med 2016; 375 (13) 1216-9.
• 3 Chennubhotla C, Clarke LP, Fedorov A, Foran D, Harris G, Helton E. et al. Survey of informatics needs in quantitative imaging for precision medicine in cancer. Yearb Med Inform. 2017
• 4 Moss TJ, Lake DE, Calland JF, Enfield KB, Delos JB, Fairchild KD. et al. Signatures of subacute potentially catastrophic illness in the ICU: model development and validation. Crit Care Med 2016; 44 (09) 1639-48.
• 5 Springer DB, Tarassenko L, Clifford GD. Logistic regression-HSMM-based heart sound segmentation. IEEE Trans Biomed Eng 2016; 63 (04) 822-32.
• 6 Hernando A, Lazaro J, Gil E, Arza A, Garzon JM, Lopez-Anton R. et al. Inclusion of respiratory frequency information in heart rate variability analysis for stress assessment. IEEE J Biomed Health Inform 2016; 20 (04) 1016-25.
• 7 Hravnak M, Chen L, Dubrawski A, Bose E, Clermont G, Pinsky MR. Real alerts and artifact classification in archived multi-signal vital sign monitoring data: implications for mining big data. J Clin Monit Comput 2016; 30 (06) 875-88.
• 8 Chincarini A, Sensi F, Rei L, Gemme G, Squarcia S, Longo R. et al. Integrating longitudinal information in hippocampal volume measurements for the early detection of Alzheimer’s disease. Neuroimage 2016; 125: 834-47.
• 9 Petousis P, Han SX, Aberle D, Bui AA. Prediction of lung cancer incidence on the low-dose computed tomography arm of the National Lung Screening Trial: A dynamic Bayesian network. Artif Intell Med 2016; 72: 42-55.
• 10 Arnold CW, Wallace WD, Chen S, Oh A, Abtin F, Genshaft S. et al. RadPath: a web-based system for integrating and correlating radiology and pathology findings during cancer diagnosis. Acad Radiol 2016; 23 (01) 90-100.
• 11 Licurse MY, Lalevic D, Zafar HM, Schnall MD, Cook TS. Expanding the scope of an automated radiology recommendation-tracking engine: initial experiences and lessons learned. J Digit Imaging 2017; 30 (02) 156-62.
• 12 Sun W, Tseng TB, Zhang J, Qian W. Enhancing deep convolutional neural network scheme for breast cancer diagnosis with unlabeled data. Comput Med Imaging Graph 2017; 57: 4-9.
• 13 Summers RM. Progress in fully automated abdominal CT interpretation. AJR Am J Roentgenol 2016; 207 (01) 67-79.
• 14 Peikari M, Gangeh MJ, Zubovits J, Clarke G, Martel AL. Triaging diagnostically relevant regions from pathology whole slides of breast cancer: a texture based approach. IEEE Trans Med Imaging 2016; 35 (01) 307-15.
• 15 Milchenko M, Snyder AZ, LaMontagne P, Shimony JS, Benzinger TL, Fouke SJ. et al. Heterogeneous optimization framework: reproducible preprocessing of multi-spectral clinical MRI for neuro-oncology imaging research. Neuroinformatics 2016; 14 (03) 305-17.
• 16 Brown JM, Horner NR, Lawson TN, Fiegel T, Greenaway S, Morgan H. et al. A bioimage informatics platform for high-throughput embryo phenotyping. Brief Bioinform. 2016
• 17 Liu C, Springer D, Li Q, Moody B, Juan RA, Chorro FJ. et al. An open access database for the evaluation of heart sound algorithms. Physiol Meas 2016; 37 (12) 2181-213.
• 18 Kalpathy-Cramer J, Zhao B, Goldgof D, Gu Y, Wang X, Yang H. et al. A comparison of lung nodule segmentation algorithms: methods and results from a multi-institutional study. J Digit Imaging 2016; 29 (04) 476-87.