Summary
Background: Federated medical search engines are health information systems that provide a single
access point to different types of information. Their efficiency as clinical decision
support tools has been demonstrated through numerous evaluations. Despite their rigor,
very few of these studies report holistic evaluations of medical search engines and
even fewer base their evaluations on existing evaluation frameworks.
Objectives: To evaluate a federated medical search engine, MedSocket, for its potential net benefits
in an established clinical setting.
Methods: This study applied the Human, Organization, and Technology (HOT-fit) evaluation framework
in order to evaluate MedSocket. The hierarchical structure of the HOT-factors allowed
for identification of a combination of efficiency metrics. Human fit was evaluated
through user satisfaction and patterns of system use; technology fit was evaluated
through the measurements of time-on-task and the accuracy of the found answers; and
organization fit was evaluated from the perspective of system fit to the existing
organizational structure.
Results: Evaluations produced mixed results and suggested several opportunities for system
improvement. On average, participants were satisfied with MedSocket searches and confident
in the accuracy of retrieved answers. However, MedSocket did not meet participants’
expectations in terms of download speed, access to information, and relevance of the
search results. These mixed results made it necessary to conclude that in the case
of MedSocket, technology fit had a significant influence on the human and organization
fit. Hence, improving technological capabilities of the system is critical before
its net benefits can become noticeable.
Conclusions: The HOT-fit evaluation framework was instrumental in tailoring the methodology for
conducting a comprehensive evaluation of the search engine. Such multidimensional
evaluation of the search engine resulted in recommendations for system improvement.
Citation: Saparova D, Belden J, Williams J, Richardson B, Schuster K. Evaluating a federated
medical search engine: Tailoring the methodology and reporting the evaluation outcomes.
Appl Clin Inf 2014; 5: 731–745
http://dx.doi.org/10.4338/ACI-2014-03-RA-0021
Keywords
Search engine - information storage and retrieval - health information systems - evaluation
- methods - family physicians
