Summary
Background: Clinical knowledge bases of problem-medication pairs are necessary for many informatics
solutions that improve patient safety, such as clinical summarization. However, developing
these knowledge bases can be challenging.
Objective: We sought to validate a previously developed crowdsourcing approach for generating
a knowledge base of problem-medication pairs in a large, non-university health care
system with a widely used, commercially available electronic health record.
Methods: We first retrieved medications and problems entered in the electronic health record
by clinicians during routine care during a six month study period. Following the previously
published approach, we calculated the link frequency and link ratio for each pair
then identified a threshold cutoff for estimated problem-medication pair appropriateness
through clinician review; problem-medication pairs meeting the threshold were included
in the resulting knowledge base. We selected 50 medications and their gold standard
indications to compare the resulting knowledge base to the pilot knowledge base developed
previously and determine its recall and precision.
Results: The resulting knowledge base contained 26,912 pairs, had a recall of 62.3% and a
precision of 87.5%, and outperformed the pilot knowledge base containing 11,167 pairs
from the previous study, which had a recall of 46.9% and a precision of 83.3%.
Conclusions: We validated the crowdsourcing approach for generating a knowledge base of problem-medication
pairs in a large non-university health care system with a widely used, commercially
available electronic health record, indicating that the approach may be generalizable
across health-care settings and clinical systems. Further research is necessary to
better evaluate the knowledge, to compare crowdsourcing with other approaches, and
to evaluate if incorporating the knowledge into electronic health records improves
patient outcomes.
Citation: McCoy AB, Wright A, Krousel-Wood M, Thomas EJ, McCoy JA, Sittig DF. Validation of
a crowdsourcing methodology for developing a knowledge base of related problem-medication
pairs. Appl Clin Inf 2015; 6: 334–344
http://dx.doi.org/10.4338/ACI-2015-01-RA-0010
Keywords
Crowdsourcing - electronic health records - knowledge bases - problem-oriented medical
records - computer-assisted drug therapy - validation studies
