A Systematic Review of Quantitative Methods for Evaluating Electronic Medication Administration Record and Bar-Coded Medication Administration Usability

Keywords

testing - evaluation - medication management - BCMA - medication administration record - interface - usability - satisfaction - quantitative

Background and Significance

Several nurse-facing technologies, such as the electronic medication administration record (eMAR) and bar-coded medication administration (BCMA), support inpatient medication administration. Integrating these medication administration technologies (MATs) into the inpatient workflow is linked with reducing medication errors.[1] [2] [3] [4] However, poor usability of MAT can negatively impact provider workflow and have unintended consequences, resulting in patient safety risks.[5] [6] A growing body of research shows the association between poor health information technology (health IT) usability and patient safety.[7] [8] [9] [10] [11] Therefore, it is important to expand assessing the impact of MAT from traditional metrics such as medication errors to broader measures that assess usability.

Usability refers to the features of a product that enable its users to achieve their goals effectively, efficiently, and satisfactorily. Effectiveness is the extent to which users can achieve their task goals with accuracy and completeness. Efficiency refers to the relative expenditure of time and other resources to achieve task goals. Satisfaction refers to user perceptions about the product, including attitudes, beliefs, and expectations about use.[12] [13] [14] [15]

There has been a steady growth in research examining MAT usability. This research is varied in theoretical frameworks, methodology, and results. Thus, there is a need to synthesize the findings of this body of research through a systematic review.

Objectives

The objective of our systematic review was to assess BCMA and eMAR usability, operationalized as efficiency, effectiveness, and satisfaction. Our specific research questions were as follows:

• What is the state of the science about the usability of BCMA and eMAR?

• What research methods and designs are used to assess BCMA and eMAR usability?

• What is the quality of the research conducted on BCMA and eMAR usability?

Methods

We used Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to evaluate the usability of studies generating quantitative data on nurse-facing MAT, specifically BCMA, and eMAR.[16] A review protocol was not published prior to beginning this review.

Search Strategy

In consultation with medical librarians, we constructed a search strategy to retrieve peer-reviewed journal articles on BCMA and eMAR usability from scientific databases (PsycInfo and MEDLINE searched from 1946 to August 20, 2019; EMBASE searched from 1974 to October 23, 2019; see [Supplementary Appendix 1] [available in the online version] for search strategy).

Inclusion Criteria

We included articles that met all of the following criteria: (1) empirical; (2) peer-reviewed journal article; (3) include a nurse-facing MAT (i.e., eMAR, BCMA); (4) if participants were recruited, the sample includes nurses; (5) data collection in an inpatient setting or its simulation; (6) measures relate to effectiveness outcomes (e.g., medication errors), work efficiency, or nurse satisfaction; (7) dependent variables (DVs) include at least one quantitative measure (e.g., counts, scores, percentages, times); (8) article published in English; and (9) article published after 1990. No articles were excluded for scientific quality because research method quality assessment was one of our primary research questions.

Data Extraction

All levels of review, data extraction, and quality assessment were performed by reviewers with advanced training in human factors engineering or usability.

Title and Abstract Review

Two reviewers independently screened 300 titles and abstracts through Covidence,[17] a software to support conducting systematic reviews, and resolved discrepancies through discussion. The inclusion/exclusion criteria evolved during this process to refine the identification of studies relevant to our review's specific research questions and objectives. Refinement broadened the included study methods from traditional human factors measures (e.g., workload evaluations) to more general measures used in patient safety research (e.g., incident reports). We also refined our exclusion criteria to exclude studies published before 1990 to evaluate the usability of MAT systems that most closely adhere to the capabilities of current systems. Outcome measures focused on adverse drug events were excluded since they were likely to capture atypical and nonrepresentative workflows. We also excluded studies that described MAT implementation without reporting on usability measures. All criteria were vetted and validated by clinical and human factors experts of the research team. Reviewers independently reviewed the remaining articles using the final review criteria before reconvening in pairs; discrepancies were resolved through discussion until consensus.

Full-Text Review

Full text reviews were conducted in two stages. First, articles that failed to meet the inclusion criteria upon full-text review were excluded. Second, two pairs of reviewers independently extracted details about the methodology and data analysis through a REDCap survey.[18] [19] For the remaining articles, we extracted the following nominal data: MAT type (i.e., BCMA or eMAR), implementation setting (e.g., country, number of institutions, departments), participant characteristics, research design, procedure, quantitative data analysis methods, results, and quantitative DVs.

When results were presented by subtasks (e.g., time to complete individual tasks) and aggregated tasks (e.g., total time to complete medication administration), we prioritized aggregated results. Additionally, when multiple results existed for a single DV (e.g., item-wise survey results), we prioritized results highlighted by authors in the abstract and discussion (e.g., aggregated survey results mapped to constructs).

Data Synthesis

After data extraction, studies with similar DVs were grouped into specific categories (e.g., medication error, task time) to facilitate the comparison of similar data. These specific categories were then grouped together based on the cardinal usability categories: effectiveness, efficiency, and/or satisfaction. Effectiveness-focused DVs measured the user's ability to leverage technology to achieve task goals, including accuracy (e.g., medication error/accuracy) and compliance with MAT after implementation. Efficiency-focused DVs measured the technology's use of resources, including task completion time and task completion metrics (e.g., task count). Satisfaction-focused DVs measured nurses' perceptions about the technology (e.g., satisfaction, perceived ease of use).[12] [13] [14] [15] [20] Articles with more than one DV could be included in more than one cardinal usability category and specific category.

Finally, results within each method were organized based on the polarity of the MAT's impact (i.e., positive, neutral, negative) on outcomes (e.g., medication error reduction is a positive impact; increase in medication administration time is a negative impact). A meta-analysis was deemed inappropriate due to the variety of methods, DVs, and their operationalizations in terms of specific measures.

Quality Assessment

We assessed research quality using a modified version of the Medical Education Research Study Quality Instrument (MERSQI).[21] Modifications included additional criteria for sampling method (e.g., randomized, snowball sampling), additional criteria for data collection reliability, and removing analysis of outcomes. We coded the research design, sampling, validity, reliability, and data analysis. Articles could score between 10 (high quality) and 1 (low quality). Two reviewers independently coded 15% of the articles (n = 6) with the modified MERSQI. After achieving a kappa of 0.71, indicating substantial inter-rater reliability, articles were coded by a single coder.

Results

[Fig. 1] shows the PRISMA chart depicting how many articles were included and excluded per review phase.

Fig. 1 PRISMA diagram illustrating screening process of identifying articles with quantitative measures of eMAR and BCMA usability

After finalizing the 41 articles, we first characterize the frequency of the MAT type and the setting of use. Next, we organize results by the three cardinal usability categories: effectiveness, efficiency, and satisfaction. Within each category, we then list the specific DVs and organize results by the study design.

Overview

[Supplementary Appendix 2] (available in the online version) contains data extracted for each study. Of the 41 articles, twenty-four (58.5%) investigated BCMA usability only, 10 (24.4%) investigated eMAR only, and seven (17.1%) investigated MAT (both BCMA and eMAR). Thirty articles (73.1%) reported unit-specific MAT usage (e.g., intensive care, medical-surgical, simulation lab, acute care, emergency department, rehab, telemetry) and specialties (e.g., oncology and surgery). Eleven articles (26.8%) reported institution-wide data.

[Fig. 2] depicts the link between three cardinal usability dimensions on the left-hand side with specific DVs (center) and data collection methods (right-hand side). The thickness of the bands indicates the number of articles measuring that specific category. Thus, effectiveness and satisfaction were used in more articles than efficiency, medication error was measured in more articles compared to other DVs, and more articles used observations and surveys compared to other methods. Articles had a median of 1 DV (range = 1–5) related to usability, resulting in a total of 100 DVs identified across the 41 final articles. Twenty-five articles (61.0%) reported one DV, six articles (14.6%) reported two, seven articles (17.1%) reported three, two articles (4.9%) reported four, and one article (2.4%) reported five DVs related to usability. Twenty-four articles (58.5%) contained DVs related to effectiveness, 8 (19.5%) to efficiency, and 17 (41.5%) to satisfaction. Five methods were used for data collection, including observations (n = 19, 46.3%), surveys (n = 17, 41.5%), patient safety event (PSE) reports (n = 9, 22.0%), BCMA surveillance data (n = 6, 14.6%), and audit (n = 3, 7.3%). Of the 100 dependent measures, approximately half of the measures of effectiveness (n = 23, 52.3%) found improvement (e.g., error reduction, increased compliance, better information accuracy) after BCMA and/or eMAR implementation, 45.5% (n = 20) of effectiveness measures found no change, and 2.27% (n = 1) of effectiveness measures found an increase in medication errors after implementation. Measures of efficiency found mixed results with 27.3% (n = 3) reporting improved efficiency after BCMA and/or eMAR implementation, 45.5% (n = 5) reporting efficiency unchanged, and 27.3% (n = 3) reporting reduced efficiency. Improved satisfaction after BCMA and/or eMAR implementation was reported in 62.2% (n = 28) of measures, while 24.4% (n = 11) and 13.3% (n = 6) reported neutral and negative outcomes, respectively.

Fig. 2 Visualization of the dependent variables by usability category and method.

Effectiveness

Twenty-four articles (58.5%) reported MAT effectiveness through the following DVs: medication errors (n = 23, 56.1%), compliance (n = 2, 8.3%), and information accuracy (n = 1, 4.2%).

Medication Error

Twenty-three of the 24 articles that reported effectiveness (95.8%) measured medication error using direct observation, electronic health record (EHR) audits, patient safety reports, and analysis of BCMA surveillance data. Eleven of the 23 articles measuring medication error (47.8%) computed medication error rate by observations by comparing observed opportunities for error with the number of errors witnessed.[22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] Nine articles (of 23, 39.1%) reported the number of PSE as a proxy for medication errors.[27] [29] [33] [34] [35] [36] [37] [38] [39] Six articles (of 23, 26.1%) reviewed BCMA surveillance data to identify medication errors via alerts.[29] [37] [40] [41] [42] [43] Two articles (of 23, 8.7%) used a prospective medical record audit to identify medication error rates.[42] [44]

Interrupted time series (ITS) design: one study (of 23, 4.3%) using this design found no significant difference in the medication error rate before and after eMAR implementation.[26]

Pretest/posttest design: seventeen articles (of 23, 73.9%) reported medication error rates before and after MAT implementation. Fourteen articles reported a reduction in the medication error rate (range = 22.5–80.7) after implementing BCMA,[22] [23] [24] [28] [30] [34] [38] [42] eMAR,[32] [36] [44] or BCMA and eMAR.[25] [29] [31] [39] One article reported a 1.12% increase in medication accuracy after MAT implementation.[31] Five articles reported no significant effect of MAT implementation on medication errors[28] [29] [33] [35] [38]; one article reported a 14.7% increase in medication error rate after BCMA implementation.[42]

Posttest-only design: seven articles (of 23, 30.4%) reported medication error rates after MAT implementation. One article reported an observed error rate of 5%. The same article also reported seven PSE reports at least 1 month after MAT implementation.[27] One article reported zero medication PSE reports being submitted before or after MAT implementation.[37] Five studies identified BCMA alert rates ranging from 0.073 to 42%.[29] [37] [41] [42] [43] One study found no significant difference between alerts using portable versus wall-mounted BCMA.[40]

Error type: eight articles (of 23, 34.8%) reporting medication error rates excluded a specific type of medication error, such as documentation errors,[24] errors involving intravenous medications,[25] wrong time errors,[28] [29] [30] [31] omission due to medication unavailability,[26] and wrong technique errors.[29] Two articles (of 23, 8.7%) identified only one medication error type, such as wrong-patient errors,[38] wrong drug errors,[38] and adverse drug events.[42] All articles found a reduction in specific types of medication errors after BCMA implementation.

Compliance

Two studies out of the 24 that reported effectiveness (8.3%) evaluated the nurse's compliance with MAT. One study using observation found that bedside use of BCMA increased 58.9% after implementation.[33] A second study using BCMA surveillance data and a posttest-only design found greater than 90% usage with wall-mounted and portable BCMA after implementation.[40]

Information Accuracy

One article out of the 24 (4.2%) measuring effectiveness through EHR audits found medication information more accurate in the eMAR than paper but did not quantify the amount by which accuracy increased.[45]

Efficiency

Eight articles (19.5%) reported efficiency-related measures of MAT. All eight articles (100%) measured the DV of task time. One article (of 8, 12.5%) measured task count, and one (of 8, 12.5%) measured scan attempts.

Task Time

All eight articles measuring efficiency (100%) used observation to measure time to complete medication administration using MAT. The studies differed in the types of tasks comprising the medication administration workflow.

Randomized controlled trial (RCT): one of the eight studies measuring task time (12.5%) found no difference in time to complete medication administration using a wall-tethered BCMA system compared to a handheld BCMA system in a simulated setting.[46]

Pretest/posttest design: five studies (of 8, 62.5%) compared task time before and after MAT implementation. One study found a 20% decrease in task time after MAT implementation.[25] Two studies did not find a significant time difference using eMAR versus paper MAR.[47] [48] Two articles reported a 27.4 and 42.0% respective increase in medication administration time after eMAR implementation.[36] [45] One study found nursing time on medication tasks outside of drug rounds increased by 36.1% after MAT implementation.[25]

Posttest-only design: two studies (of 8, 25.0%) compared hospital unit outcomes after BCMA implementation. One article reported that nurses using BCMA spent 30.4% less time on medication-related tasks than the non-BCMA control group.[49] One study found no significant time difference when administering medications with portable versus wall-mounted BCMA.[40]

Task Count

One observation study out of eight measuring efficiency (12.5%) found that the average number of medication tasks reduced by 51.2% after eMAR implementation.[47]

Scan Attempts

One study out of eight measuring efficiency (12.5%) found no significant difference in scan attempts with a wall-tethered versus a handheld BCMA system.[46]

Satisfaction

Seventeen articles (41.5%) reported satisfaction-related measures. The most common DV was general satisfaction, measured by 8 of 17 studies (47.1%). Seven studies measured perceived ease of use (of 17, 41.2%), five measured perceived timeliness (of 17, 29.4%), five measured perceived usefulness (of 17, 29.4%), five measured safety (of 17, 29.4%), three measured behavioral intention (of 17, 17.6%), and two measured workload (of 17, 11.8%).

All 17 studies used surveys to measure their DVs. See [Table 1] for details. Since many studies used Likert scale surveys that varied in constructs and response options, we could not quantify the overall change in the construct measured unless reported by the study. Six studies (of 17, 35.3%) used validated surveys, including the Technology Acceptance Model (TAM) questionnaire,[50] [51] the Perceived Social Influence (SI) questionnaire,[52] the Medication Administration System-Nurses Assessment of Satisfaction (MAS-NAS),[53] the Questionnaire for User Interaction Satisfaction,[54] [55] and the Post Study System Usability Questionnaire (PSSUQ).[56] Eleven studies (of 17, 64.7%) used novel surveys. Two articles (of 17, 11.8%) did not describe the surveys used for the study.

Table 1 Survey constructs, validation metrics, and reliability metrics of included articles

Survey used	Constructs	Article	Validity	Reliability
Technology Acceptance Model (TAM)[50] [51]	Perceived ease of use (PEOU) Perceived usefulness (PU) Behavioral intention to use (BI)	Darawad et al[59] Song et al[66] Holden et al[62]	Generated by a literature search Evaluated using cognitive interviewing and categorization tasks	Cronbach's alpha: Ease of use: 0.94 Usefulness: 0.98
Medication Administration System-Nurses Assessment of Satisfaction (MAS-NAS)[53]	Satisfaction (efficacy, safety, access)	Darawad et al[59] Hurley et al[58]	Evaluated by an expert review and focus group Pilot tested Factors analysis	Cronbach's alpha: 0.86
Perceived Social Influence (SI)[52]	Perceived social influence (SI) (also known as subjective norms, SN)	Holden et al[62]	Validity scale assessment	Cronbach's alpha: 0.86
Post Study System Usability Questionnaire (PSSUQ)[56]	System usefulness Information quality Interface quality	Landman et al[46]	Generated by usability experts Factors analysis	Coefficient alpha: 0.97 Subscale coefficient alpha: 0.91–0.96
Questionnaire for User Interaction Satisfaction[54] [55]	General satisfaction Screen Terminology and system information Learning System capabilities	Staggers et al[61]	Factors analysis	Cronbach's alpha: 0.94 Interitem alpha values: 0.93–0.94
Darawad, Othman, & Alosta Novel Survey	Satisfaction with received BCMA training Level of comfort while using the BCMA Level of competency using a computer at work Perception of job productivity enhanced by BCMA Overall rating of BCMA by nurses	Darawad et al[59]	Not described	Not described
Dasgupta et al Novel Survey	Perceptions of workflow ease Perceptions of barriers during medication administration	Dasgupta et al[64]	Generated by a literature search	Not described
Gaucher & Greer Novel Survey	Nurses' attitudes toward the unit dose system and computerized MAR	Gaucher and Greer	Developed by a pharmacist and nurse Pilot tested	Not described
Holden et al 2011 Novel Survey	Perceptions of: Accuracy Usefulness Consistency Time efficiency Ease of Performance Error likelihood Error detection likelihood	Holden et al[63]	Evaluated using expert review and cognitive interviewing	0.92 (reported in Karsh et al)
Holden et al 2012 Novel Survey	Perceived usefulness for patient care Perceived social influence from patient/family Satisfaction	Holden et al[62]	Evaluated using cognitive interviewing Average variance: 0.26	Cronbach's alpha: 0.73
Holden et al 2015 Novel Survey	Perceptions of: External mental workload Internal mental workload Medication administration error likelihood Unit level medication error likelihood Unit level ADE likelihood	Holden et al[69]	Evaluated using cognitive interviewing	Cronbach's alpha: 0.71 and 0.72
Lin, Lee, & Mills Novel Survey	System quality Information quality Service quality Overall user satisfaction Usage benefits	Lin et al[34]	Content validity index Relevance: 0.90 Text clarity: 0.84	Cronbach's alpha: 0.85
Ludwig-Beymer et al Novel Survey	Medication administration Patient care Ease of use Computer availability Reliability of technology	Ludwig-Beymer et al[40]	Evaluated using expert review Content validity index: 0.94	Survey completed twice by same nurses 2 weeks apart
Maydana et al Novel Survey	Ease of use Weight of scanner System utility Interference with patient care Training Support Level of satisfaction with the system Suggestions	Maydana et al[65]	Not described	Not described
Moreland et al Novel Survey	Nurse workload Teamwork Ease of documentation Drug interactions Accuracy Patient safety Satisfaction	Moreland et al[57]	Content validity index: 0.92	Cronbach's alpha: 0.92
Morriss et al Novel Survey	Learning to use the system Nurses' opinions of effectiveness of the BCMA system Nurses' opinions of the alerts issued by the system Effect of the BCMA System on nursing: Workflow Workarounds Professionalism Job satisfaction Diffusion of innovation Acceptance of technology	Morriss et al[60]	Not described	Not described
Not described	Not described	Mitchell et al[45] Tsai et al	Not described	Not described

Abbreviations: ADE, adverse drug event; BCMA, bar-coded medication administration; MAR, medication administration record.

General Satisfaction

Eight of the 17 articles (47.1%) measured nurses' general satisfaction with MAT.

Pretest/posttest design: two of the eight studies measuring general satisfaction (25.0%) found increased nurse satisfaction after eMAR[57] and MAT implementation.[58] One study found a positive association between eMAR documentation and nurse satisfaction.[57]

Posttest-only design: six studies (of 8, 75.0%) investigated nurses' satisfaction after MAT implementation, of which four found increased nurse's satisfaction.[45] [59] [60] [61] One study found that perceived satisfaction was predicted by perceived ease of use, perceived usefulness for the patient, and perceived social pressure from patients and families to use BCMA.[62] Another study found information quality of BCMA predicted satisfaction.[34] Only one article reported low nurse satisfaction with BCMA.[62]

Perceived Ease of Use

Seven of the 17 articles (41.2%) measured nurses' perceived usefulness of BCMA.

RCT: one of the seven studies that measured perceived ease of use (14.3%) found that 95% of nurses perceived a handheld BCMA system as easy to use in a simulated setting.[46]

Pretest/posttest design: one study (of 7, 14.3%) found that nurses' perceptions about the ease of documentation decreased after BCMA implementation.[63]

Posttest-only design: five studies (of 7, 71.4%) evaluated BCMA's perceived ease of use only after implementation. Three articles reported that nurses perceived BCMA as easy to use.[62] [64] [65] One article reported that “feedback about errors” positively influenced BCMA ease of use.[66] One study found no difference between the ease of use of portable BCMA versus wall-mounted BCMA.[40]

Perceived Timeliness

Five of the 17 articles (29.4%) measured nurses' perceived usefulness of MAT. Two of the five studies that measured perceived timeliness (40.0%) used a pretest/posttest design. One article reported that most nurses believed BCMA would reduce working times.[67] One study found decreased perceptions of documentation time efficiency with BCMA after implementation.[63] Three studies (of 5, 60.0%) used a posttest-only design: one article reported that most nurses believed eMAR would reduce working times[68]; a second study found that about half of the nurses strongly agreed that BCMA improved the timeliness of medication[64]; the third study found that most nurses believed BCMA required more time than their previous paper system.[60]

Perceived Usefulness

Five of the 17 articles (29.4%) measured the perceived usefulness of BCMA. Two of the five studies (40.0%) using a pretest/posttest design found that nurses perceived BCMA as a useful tool that helped with patient care.[63] [67] One article reported that nurses negatively perceived usefulness of BCMA regarding medication documentation.[63] Three studies (of 5, 60.0%) used a posttest-only design. One article reported that the usefulness of BCMA was predicted by “feedback about communication and errors.”[66] One article reported that permanent bedside BCMAs were more readily available than portable BCMA.[40] One article reported that nurses negatively perceived BCMA's overall usefulness.[62]

Perceived Safety

Five of the 17 articles (29.4%) measured nurses' perceived safety of MAT. Three of the five studies (60.0%) using a pretest/posttest design found that nurses perceived improved care safety with BCMA.[63] [67] [69] One article reported an association between high external workload and perceived medication safety events, and that improvements to perceived medication safety decayed to the pre-BCMA level over time.[69] Two articles using a posttest-only design (of 5, 40.0%) reported that nurses perceived improved care safety with the introduction of BCMA or eMAR.[60] [68]

Behavioral Intention to Use

Three of the 17 articles (17.6%) using a posttest-only design measured nurses' behavioral intention to use MAT. Two articles reported that nurses had a high behavioral intention to use BCMA.[62] [68] Two articles reported constructs associated with behavioral intention to use BCMA, including perceived ease of use,[62] perceived social influence,[62] unit teamwork,[66] perceived usefulness for patient care,[62] and perceived usefulness more generally.[66]

Perceived Workload

Two of the 17 articles (11.8%) measured nurses' perceived workflow barriers using BCMA. One of the two studies (50.0%) using a pre/post design found BCMA increased external mental workload (e.g., interruptions, divided attention, and being rushed during medication management tasks) but decreased internal mental workload (e.g., requirements for concentration and mental effort during medication management tasks) after implementation.[69] The other posttest-only study (of 2, 50.0%) found that half of the nurses strongly agreed that BCMA resulted in less workload.[64]

Quality Scores

The median quality score was 5.50 (range = 1.50–8.25). Articles with DVs related to effectiveness had the highest scores (median = 5.50; range = 3.00–8.00), followed by efficiency (median = 5.20; range = 3.00–8.25), then satisfaction (median = 5; range = 1.25–8.25).

Research Design

A weak research design was the main reason for low-quality scores. [Fig. 3] shows the research designs classified by the cardinal usability category. Researchers frequently use pretest/posttest designs, but only a minority of studies use the more rigorous research design of the RCT. One study used an RCT design (2.4%), and another used an ITS design (2.4%). Twenty-one articles (51.2%) used a pretest/posttest design, of which 11 (of 21, 52.4%) measured a single group, eight (of 21, 38.1%) measured two or more nonrandomized groups, and one (of 21, 4.8%) measured a single group for one DV and two or more nonrandomized groups for another DV. Fourteen articles (34.1%) used a posttest-only design, of which eight (of 14, 57.1%) measured a single group and six (42.9%) measured two or more nonrandomized groups. Four articles (9.8%) used pretest/posttest and posttest-only designs for different DVs.

Fig. 3 Number of distinct articles by usability category and study design.

Sampling Procedure

Many studies had generalizability limitations due to setting (35 articles [85.4%] conducted at a single institution) and sampling methodologies (i.e., convenience sampling).

Validity of Data Collection

Validity of data collection instruments was variable and dependent on the data collection method. Observation and audit methods typically had clearly defined variables and structured data collection forms. Five of the seventeen (29.4%) studies using surveys did not report content, criterion, or construct validity of the survey tool. Multiple articles did not differentiate true BCMA alerts from errant alerts. PSE reports typically represent an underestimation of any institution's total number of errors.

Reliability of Data Collection

The reliability of the data collection methods was poor across most studies. Observations, audits, and surveys developed by the studies' authors sometimes failed to report reliability metrics (e.g., observer training, Cohen's kappa, Cronbach's alpha). Seven of the 17 (41.2%) survey studies did not report the reliability of the survey instrument. The reliability of BCMA data was not evaluated against other sources. None of the articles using PSE reports described reliability calculation procedures for event classification, selecting and categorizing MAT-relevant errors from PSE reports (e.g., medication stocking errors that occurred in pharmacy should not be included in MAT error counts).

Data Analysis

Data analysis methods were highly variable across studies. Many studies measuring continuous variables through observations and EHR audits favored using statistics for nominal data over more appropriate inferential statistics. Many methods (e.g., PSE reports, BCMA surveillance data) were exclusively summarized through descriptive statistics.

Discussion

Although we initially extracted 1,922 articles, our systematic screening approach resulted in only 41 studies reaching the stage of data extraction based on quantitative operationalization of the cardinal usability categories (e.g., effectiveness, efficiency, satisfaction) of their DVs. This highlights that although research may be tagged with variables approximating constructs of usability, studies that perform the necessary operationalization of usability constructs may be limited. The research on BCMA usability (n = 24 studies) is extensive compared to eMAR (n = 10) and BCMA and eMAR (n = 7). A plurality of the research investigates usability DVs related to effectiveness (n = 24) and satisfaction (n = 17); only a handful of studies investigate efficiency (n = 8). Articles used five methods: observation, audit logs, PSE reports, BCMA surveillance, and surveys; however, there was substantial variation in how data collection methods were implemented across articles.

Many findings mirror those from previous systematic reviews.[70] [71] [72] [73] [74] [75] [76] [77] [78] [79] Previous reviews found minimal published research on MAT, especially compared to the extensive research on physician-facing technologies like computerized physician order entry. More reviews focused on BCMA technology rather than eMAR or other nurse-facing health IT, and most reviews focused on the impact of MAT on medication error. In contrast, there has been lesser attention paid to how MAT design affects its intended use and the important impact of MAT usability on nursing workflow and satisfaction.

Recommendations for Future Research Focus

The outcomes of quantitative measures of effectiveness, efficiency, and satisfaction are varied. Approximately half of the studies measuring effectiveness found a reduction in medication error rates after MAT implementation. Measures of efficiency, such as task time, found mixed results. Measures of satisfaction indicated largely positive or neutral perceptions of MAT. Through these varied methods and results, we identified the following areas of focus for future research.

Increase Studies Focusing on eMAR

Articles focused on BCMA were more than double those investigating eMAR. In contrast to BCMA, eMAR may be perceived as an electronic interpretation of a long-familiar health care tool: the paper MAR. However, porting a paper tool into an electronic form is not simple and can lead to unintended consequences. More research is needed on eMAR usability to identify these unintended consequences and optimize the efficiency of existing eMAR technologies.

Increase Research on MAT Efficiency

Another noteworthy finding is that 89% of the articles focus on the usability categories of effectiveness and satisfaction. Although these foci made sense in the landscape where the health care community needed to know whether these new technologies could perform their intended tasks, it is becoming clear that inefficiencies associated with using health IT, such as prolonged documentation time and the time cost in recovering from automation surprises,[80] [81] [82] can contribute to clinician frustration and burnout.[11] [83] [84] Thus, there is an increasing need to shift the research focus from effectiveness to efficiency to address clinician cognitive needs in medication management.[85] [86]

Increase Use of Novel Research Methods in Usability

Quantitative MAT research uses five primary methods of data collection: observation, audit, PSE reports, BCMA surveillance, and survey. There is tremendous scope to broaden the range of research questions and quantitative metrics about MAT usability through novel methods such as eye tracking,[83] physiological response (e.g., heart rate),[87] and user interaction data (e.g., keyboard and mouse input).[88]

Investigate the Effect of MAT Usability on Patient Outcomes

This systematic review focused on the nurse-facing usability of MAT. While research shows poor usability can lead to patient safety issues,[7] [8] [9] this systematic review did not investigate the impact of usability issues on patient outcomes. Future research should investigate the impact of poor MAT usability on patient outcomes.

Improve Rigor of Research Methods

Similar to previous systematic reviews on MAT and medication errors, we found wide variation in research designs used by the studies included in our review. Only one out of the 41 articles used a RCT. Pretest/posttest designs are currently most frequently used in MAT evaluations. Although the pretest/posttest design is a good starting point, future research should attempt to control the selection and assignment of participants to study conditions more systematically and control confounding variables through the use of RCTs.

Standardize Methodological Reporting to Ensure Interpretation and Replication

Our review found wide variability in the operationalization, measurement, and reporting of methodological and procedural details. This variability complicates the direct comparison between studies that is essential for a meta-analysis and makes replication challenging. Definitions and measures of medication management tasks vary for the same study method and across studies. For example, there may be variation in the definition of medication error, with some studies counting errors precluded because of using MAT and some studies excluding errors caught by MAT. There is also considerable ambiguity in decomposing the impact of medication errors on patient outcomes in terms of harm events versus near-misses. In addition, many studies do not report the interval between implementation and measurement and the duration of data collection, making it difficult to understand the effects of MAT on short- and long-term usage metrics. Studies also used different definitions for clinical processes, making it difficult to compare findings. For example, one study began medication administration total time by starting the clock when the nurse entered the patient's room.[46] However, another study also counted tasks outside the patient room (e.g., reviewing medications).[47] Researchers should develop and test standardized use cases to accomplish critical and routine tasks on the BCMA and eMAR to facilitate comparison across studies.

Define MAT Design Requirements

BCMA and eMAR are nearly ubiquitous in health care, so research needs to optimize these technologies. Optimization strategies could include standardized functional and design requirements, a common practice with medication order entry technologies that do not currently exist for MAT. Most articles do not describe their EHR vendor and MAT design customizations at individual sites, making it difficult to link specific design elements with MAT usability. Because the design of MAT is integral to its usability and success, future research should describe the EHR vendor, version, and customizations of MAT, and move toward building critical functional requirements to optimize MAT's effectiveness, efficiency, and satisfaction.

Policy Implications

Lessons learned from this review have policy implications. First, federal agencies and other organizations funding research on MAT may want to consider encouraging researchers to use similar measurement methods and provide greater methodological detail. In addition, some of these best practices may inform usability testing requirements that the Food and Drug Administration and Office of the National Coordinator for Health Information Technology already have in place for certain medical devices and health IT. Those organizations overseeing medical devices and health IT could identify environments in which the benefits of MAT are being realized and seek to share those results more broadly with other health care facilities.

Limitations

This systematic review should be interpreted with the following limitations: (1) The research on MAT usability is conducted with diverse methods that yield quantitative, numeric data and qualitative, thematic data. We intentionally chose to limit our review to studies yielding quantitative data because it was difficult to summarize quantitative and qualitative data together meaningfully. However, future studies should synthesize MAT research using qualitative methods such as interviews and usability evaluations to investigate how important usability facets affect workflow and perceptions about the technology. (2) Our results may be constrained by our search strategy and databases. (3) The studies used in this review did not uniformly describe at which point in the MAT implementation cycle measures were obtained (i.e., immediately after implementation vs. delayed measures). Consequently, direct comparisons between study outcomes should be made with caution. (4) Most articles included in this review contain data from settings based in the United States, potentially limiting the generalizability of our results to other countries. (5) Our review identified a variety of research designs, sampling strategies, dependent measures, and their operationalizations. Assessments of study quality in traditional medical research do not sufficiently capture this type of variation that generally occurs in studies of health IT usability. Therefore, we modified an existing tool, the MERSQI for quality assessments. Future research should develop tools that can suitably capture the broad variety of research in health IT evaluation and address quality. (6) This systematic review focuses on the nurse-facing usability of MAT. Future research will need to be performed to understand other clinicians who use MAT and the impact of MAT usability on patient outcomes.

Conclusion

This systematic review aimed to identify the state of the science, study methods, and quality of articles investigating BCMA and eMAR usability. The 41 included articles primarily focus on BCMA and measures of effectiveness and satisfaction. Study quality varies substantially. Future research should focus on eMAR usability, with a specific focus on efficiency, and should expand research designs and methods that generate generalizable MAT design requirements.

Clinical Relevance Statement

Unusable technologies increase work time, create unnecessary redundancies, and lead to clinician burnout. As the primary users of MAT, nurses take on the burden when eMAR and BCMA are not designed to be usable. This research identifies current gaps in the literature, including MAT efficiency, eMAR usability and use of novel quantitative methodologies, and identifies improvements that can be made to research reporting.

Multiple-Choice Questions

1. Which of the following are consequences of poor usability in medication administration technologies (MAT)?

   • Patient safety risks

   • Increased task time

   • Clinician burnout

   • All of the above

   Correct Answer: The correct answer is option d. All of the above. Usability refers to a product's features that enable users to achieve their goals effectively, efficiently, and satisfactorily. Although MAT is designed to increase patient safety, poor usability can lead to patient safety risks, especially if the technology creates barriers to work tasks and users must implement workarounds. Poor usability can also create inefficiencies when using MAT, such as increased documentation time and time lost recovering from automation surprises. Prior research shows that the time and frustration associated with unusable technologies contribute to clinician burnout.

2. In the quantitative articles reviewed, what was the most common dependent variable?

   • Medication error

   • Task time

   • Task count

   • General satisfaction

   Correct Answer: The correct answer is option a. Medication error. Twenty-three studies measured medication error, eight measured task time, one measured task count, and eight measured general satisfaction. In the discussion, we suggest that medication error has historically been the most important metric for MAT success as it was recently a new, minimally tested technology. Satisfaction measures may also be common, as surveys are much easier and cheaper investigation methods when compared to observation. Task time and task count comprise the smallest category of quantitative dependent variables, suggesting a gap in current literature and a direction for future research.

• References

• 1 Coyle GA, Heinen M. Evolution of bcma within the department of veterans affairs. Nurs Adm Q 2005; 29 (01) 32-38
• 2 Agrawal A. Medication errors: prevention using information technology systems. Br J Clin Pharmacol 2009; 67 (06) 681-686
• 3 Marini SD, Hasman A. Impact of BCMA on medication errors and patient safety: a summary. Stud Health Technol Inform 2009; 146: 439-444
• 4 T H, Heelon M, Siano B. et al. Medication safety improves after implementation of positive patient identification. Appl Clin Inform 2010; 1 (03) 213-220
• 5 Staggers N, Elias BL, Hunt JR, Makar E, Alexander GL. Nursing-centric technology and usability a call to action. Comput Inform Nurs 2015; 33 (08) 325-332
• 6 Ratwani RM, Reider J, Singh H. A decade of health information technology usability challenges and the path forward. JAMA 2019; 321 (08) 743-744
• 7 Ratwani RM, Savage E, Will A. et al. Identifying electronic health record usability and safety challenges in pediatric settings. Health Aff (Millwood) 2018; 37 (11) 1752-1759
• 8 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
• 9 Koppel R, Wetterneck T, Telles JL, Karsh BT. Workarounds to barcode medication administration systems: their occurrences, causes, and threats to patient safety. J Am Med Inform Assoc 2008; 15 (04) 408-423
• 10 Alexander G, Staggers N. A systematic review of the designs of clinical technology: findings and recommendations for future research. ANS Adv Nurs Sci 2009; 32 (03) 252-279
• 11 Fraczkowski D, Matson J, Lopez KD. Nurse workarounds in the electronic health record: an integrative review. J Am Med Inform Assoc 2020; 27 (07) 1149-1165
• 12 Weir CR, Taber P, Taft T, Reese TJ, Jones B, Del Fiol G. Feeling and thinking: can theories of human motivation explain how EHR design impacts clinician burnout?. J Am Med Inform Assoc 2021; 28 (05) 1042-1046
• 13 ISO/TC 159/SC 4 Ergonomics of human-system interaction. ISO - ISO 9241–11:2018 - Ergonomics of human-system interaction—Part 11: Usability: Definitions and concepts. International Organization for Standardization. Published online 2018. Accessed June 13, 2022 at: https://www.iso.org/standard/63500.html
• 14 Bevan N, Carter J, Earthy J, Geis T, Harker S. New ISO standards for usability, usability reports and usability measures. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2016. 9731. 268-278
• 15 Usability 101: Introduction to usability. Accessed June 22, 2022 at: https://www.nngroup.com/articles/usability-101-introduction-to-usability/
• 16 Page MJ, McKenzie JE, Bossuyt PM. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021; 372: n71
• 17 Veritas Health Innovation. Covidence systematic review software. Accessed June 13, 2022 at: www.covidence.org
• 18 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42 (02) 377-381
• 19 Harris PA, Taylor R, Minor BL. et al; REDCap Consortium. The REDCap consortium: building an international community of software platform partners. J Biomed Inform 2019; 95: 103208
• 20 Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422
• 21 Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA 2007; 298 (09) 1002-1009
• 22 Bonkowski J, Weber RJ, Melucci J, Pesavento T, Henry M, Moffatt-Bruce S. Improving medication administration safety in solid organ transplant patients through barcode-assisted medication administration. Am J Med Qual 2014; 29 (03) 236-241
• 23 Bonkowski J, Carnes C, Melucci J. et al. Effect of barcode-assisted medication administration on emergency department medication errors. Acad Emerg Med 2013; 20 (08) 801-806
• 24 DeYoung JL, Vanderkooi ME, Barletta JF. Effect of bar-code-assisted medication administration on medication error rates in an adult medical intensive care unit. Am J Health Syst Pharm 2009; 66 (12) 1110-1115
• 25 Franklin BD, O'Grady K, Donyai P, Jacklin A, Barber N. The impact of a closed-loop electronic prescribing and administration system on prescribing errors, administration errors and staff time: a before-and-after study. Qual Saf Health Care 2007; 16 (04) 279-284
• 26 Jheeta S, Franklin BD. The impact of a hospital electronic prescribing and medication administration system on medication administration safety: an observational study. BMC Health Serv Res 2017; 17 (01) 547
• 27 Hardmeier A, Tsourounis C, Moore M, Abbott WE, Guglielmo BJ. Pediatric medication administration errors and workflow following implementation of a bar code medication administration system. J Healthc Qual 2014; 36 (04) 54-61 , quiz 61–63
• 28 Helmons PJ, Wargel LN, Daniels CE. Effect of bar-code-assisted medication administration on medication administration errors and accuracy in multiple patient care areas. Am J Health Syst Pharm 2009; 66 (13) 1202-1210
• 29 Paoletti RD, Suess TM, Lesko MG. et al. Using bar-code technology and medication observation methodology for safer medication administration. Am J Health Syst Pharm 2007; 64 (05) 536-543
• 30 Poon EG, Keohane CA, Yoon CS. et al. Effect of bar-code technology on the safety of medication administration. N Engl J Med 2010; 362 (18) 1698-1707
• 31 Seibert HH, Maddox RR, Flynn EA, Williams CK. Effect of barcode technology with electronic medication administration record on medication accuracy rates. Am J Health Syst Pharm 2014; 71 (03) 209-218
• 32 Vicente Oliveros N, Gramage Caro T, Pérez Menendez-Conde C. et al. Effect of an electronic medication administration record application on patient safety. J Eval Clin Pract 2017; 23 (04) 888-894
• 33 Bowers AM, Goda K, Bene V. et al. Impact of bar-code medication administration on medication administration best practices. Comput Inform Nurs 2015; 33 (11) 502-508
• 34 Lin JC, Lee TT, Mills ME. Evaluation of a barcode medication administration information system. Comput Inform Nurs 2018; 36 (12) 596-602
• 35 Low DK, Belcher JV. Reporting medication errors through computerized medication administration. Comput Inform Nurs 2002; 20 (05) 178-183
• 36 McComas J, Riingen M, Chae Kim S. Impact of an electronic medication administration record on medication administration efficiency and errors. Comput Inform Nurs 2014; 32 (12) 589-595
• 37 Miller DF, Fortier CR, Garrison KL. Bar code medication administration technology: characterization of high-alert medication triggers and clinician workarounds. Ann Pharmacother 2011; 45 (02) 162-168
• 38 Sakushima K, Umeki R, Endoh A, Ito YM, Nasuhara Y. Time trend of injection drug errors before and after implementation of bar-code verification system. Technol Health Care 2015; 23 (03) 267-274
• 39 Truitt E, Thompson R, Blazey-Martin D, NiSai D, Salem D. Effect of the implementation of barcode technology and an electronic medication administration record on adverse drug events. Hosp Pharm 2016; 51 (06) 474-483
• 40 Ludwig-Beymer P, Williams P, Stimac E. Comparing portable computers with bedside computers when administering medications using bedside medication verification. J Nurs Care Qual 2012; 27 (04) 288-298
• 41 Mahoney CD, Berard-Collins CM, Coleman R, Amaral JF, Cotter CM. Effects of an integrated clinical information system on medication safety in a multi-hospital setting. Am J Health Syst Pharm 2007; 64 (18) 1969-1977
• 42 Morriss Jr FH, Abramowitz PW, Nelson SP. et al. Effectiveness of a barcode medication administration system in reducing preventable adverse drug events in a neonatal intensive care unit: a prospective cohort study. J Pediatr 2009; 154 (03) 363-368 , 368.e1
• 43 Sakowski J, Leonard T, Colburn S. et al. Using a bar-coded medication administration system to prevent medication errors in a community hospital network. Am J Health Syst Pharm 2005; 62 (24) 2619-2625
• 44 Sullins AK, Richard AA, Manasco KB, Phillips MS, Gomez TA. Which comes first CPOE or eMAR? A retrospective analysis of health information technology implementation. Journal Indexing and Metrics 2012; 47 (11) 863-870
• 45 Mitchell D, Usher J, Gray S. et al. Evaluation and audit of a pilot of electronic prescribing... - Google Scholar. J Inf Technol Healthc 2004; 2 (01) 19-29
• 46 Landman A, Neri PM, Robertson A. et al. Efficiency and usability of a near field communication-enabled tablet for medication administration. JMIR Mhealth Uhealth 2014; 2 (02) e26
• 47 Leung M, Chan KKC, Wong WL, Law ACB. Impact of IPMOE on nursing tasks in the medical ward: a time-motion study. Int J Nurs Sci 2018; 5 (01) 50-56
• 48 Westbrook JI, Li L, Georgiou A, Paoloni R, Cullen J. Impact of an electronic medication management system on hospital doctors' and nurses' work: a controlled pre-post, time and motion study. J Am Med Inform Assoc 2013; 20 (06) 1150-1158
• 49 Huang HY, Lee TT. Impact of bar-code medication administration on nursing activity patterns and usage experience in Taiwan. Comput Inform Nurs 2011; 29 (10) 554-563
• 50 Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer technology: a comparison of two theoretical models. Manage Sci 1989; 35 (08) 982-1003
• 51 Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. Manage Inf Syst Q 1989; 13 (03) 319-339
• 52 Mathieson K. Predicting user intentions: comparing the technology acceptance model with the theory of planned behavior. Inf Syst Res 1991; 2 (03) 173-191
• 53 Hurley AC, Lancaster D, Hayes J. et al. The Medication Administration System–Nurses Assessment of Satisfaction (MAS-NAS) scale. J Nurs Scholarsh 2006; 38 (03) 298-300
• 54 Chin JP, Diehl VA, Norman KL. Development of an instrument measuring user satisfaction of the human-computer interface. Paper presented at: CHI '88 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Washington, DC, United States, May 15–19, 1988; Part F130202: 213-218
• 55 Harper BD, Norman KL. Improving user satisfaction: the questionnaire for user interaction satisfaction version 5.5. Paper presented at: Proceedings of the 1st Annual Mid-Atlantic Human Factors Conference, Virginia Beach, VA, United States. 1993: 224-228
• 56 Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact 1995; 7 (01) 57-78
• 57 Moreland PJ, Gallagher S, Bena JF, Morrison S, Albert NM. Nursing satisfaction with implementation of electronic medication administration record. Comput Inform Nurs 2012; 30 (02) 97-103
• 58 Hurley AC, Bane A, Fotakis S. et al. Nurses' satisfaction with medication administration point-of-care technology. J Nurs Adm 2007; 37 (7–8): 343-349
• 59 Darawad MW, Othman EH, Alosta MR. Nurses' satisfaction with barcode medication-administration technology: results of a cross-sectional study. Nurs Health Sci 2019; 21 (04) 461-469
• 60 Morriss Jr FH, Abramowitz PW, Carmen L, Wallis AB. “Nurses Don't Hate Change” – survey of nurses in a neonatal intensive care unit regarding the implementation, use and effectiveness of a bar code medication administration system. Healthc Q 2009; 12 Spec No Patient: 135-140
• 61 Staggers N, Kobus D, Brown C. Nurses' evaluations of a novel design for an electronic medication administration record. Comput Inform Nurs 2007; 25 (02) 67-75
• 62 Holden RJ, Brown RL, Scanlon MC, Karsh BT. Modeling nurses' acceptance of bar coded medication administration technology at a pediatric hospital. J Am Med Inform Assoc 2012; 19 (06) 1050-1058
• 63 Holden RJ, Brown RL, Alper SJ, Scanlon MC, Patel NR, Karsh BT. That's nice, but what does IT do? Evaluating the impact of bar coded medication administration by measuring changes in the process of care. Int J Ind Ergon 2011; 41 (04) 370-379
• 64 Dasgupta A, Sansgiry SS, Jacob SM, Frost CP, Dwibedi N, Tipton J. Descriptive analysis of workflow variables associated with barcode-based approach to medication administration. J Nurs Care Qual 2011; 26 (04) 377-384
• 65 Maydana T, Giraldo L, González Z. et al. BarCode medication administration in ICU: learning from our nurses. Stud Health Technol Inform 2017; 245: 1095-1098
• 66 Song L, Park B, Oh KM. Analysis of the technology acceptance model in examining hospital nurses' behavioral intentions toward the use of bar code medication administration. Comput Inform Nurs 2015; 33 (04) 157-165
• 67 Tsai SL, Sun YC, Taur FM. Comparing the working time between Bar-Code Medication Administration system and traditional medication administration system: an observational study. Int J Med Inform 2010; 79 (10) 681-689
• 68 Gaucher M, Greer M. A nursing evaluation of unit dose and computerized medication administration records. Can J Hosp Pharm 1992; 45 (04) 145-150
• 69 Holden RJ, Brown RL, Scanlon MC, Rivera AJ, Karsh BT. Micro- and macroergonomic changes in mental workload and medication safety following the implementation of new health IT. Int J Ind Ergon 2015; 49: 131-143
• 70 Young J, Slebodnik M, Sands L. Bar code technology and medication administration error. J Patient Saf 2010; 6 (02) 115-120
• 71 Shah K, Lo C, Babich M, Tsao NW, Bansback NJ. Bar Code Medication Administration Technology: a systematic review of impact on patient safety when used with computerized prescriber order entry and automated dispensing devices. Can J Hosp Pharm 2016; 69 (05) 394-402
• 72 Zheng WY, Lichtner V, Van Dort BA, Baysari MT. The impact of introducing automated dispensing cabinets, barcode medication administration, and closed-loop electronic medication management systems on work processes and safety of controlled medications in hospitals: a systematic review. Res Social Adm Pharm 2021; 17 (05) 832-841
• 73 Williams R, Aldakhil R, Blandford A, Jani Y. Interdisciplinary systematic review: does alignment between system and design shape adoption and use of barcode medication administration technology?. BMJ Open 2021; 11 (07) e044419
• 74 Moore EC, Tolley CL, Bates DW, Slight SP. A systematic review of the impact of health information technology on nurses' time. J Am Med Inform Assoc 2020; 27 (05) 798-807
• 75 Hutton K, Ding Q, Wellman G. The effects of Bar-coding technology on medication errors: a systematic literature review. J Patient Saf 2021; 17 (03) e192-e206
• 76 Hassink J, Jansen M, Helmons P. Effects of bar code-assisted medication administration (BCMA) on frequency, type and severity of medication administration errors: a review of the literature. Eur J Hosp Pharm Sci Pract 2012; 19 (05) 489-494
• 77 Strudwick G, Reisdorfer E, Warnock C. et al. Factors associated with Barcode Medication Administration technology that contribute to patient safety: an integrative review. J Nurs Care Qual 2018; 33 (01) 79-85
• 78 Stolic S, Ng L, Sheridan G. Electronic medication administration records and nursing administration of medications: an integrative review. Nurse Education Today 2022; 105325
• 79 Khammarnia M, Kassani A, Eslahi M. The efficacy of patients' wristband bar-code on prevention of medical errors: a meta-analysis study. Appl Clin Inform 2015; 6 (04) 716-727
• 80 Patterson ES. Workarounds to intended use of health information technology: a narrative review of the human factors engineering literature. Hum Factors 2018; 60 (03) 281-292
• 81 Patterson ES, Cook RI, Render ML. Improving patient safety by identifying side effects from introducing bar coding in medication administration. J Am Med Inform Assoc 2002; 9 (05) 540-553
• 82 Carayon P, Wetterneck TB, Hundt AS. et al. Evaluation of nurse interaction with bar code medication administration technology in the work environment. J Patient Saf 2007; 3 (01) 34-42
• 83 Asan O, Yang Y. Using eye trackers for usability evaluation of health information technology: a systematic literature review. JMIR Human Factors 2015; 2 (01) e5
• 84 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
• 85 Belden JL, Wegier P, Patel J. et al. Designing a medication timeline for patients and physicians. J Am Med Inform Assoc 2019; 26 (02) 95-105
• 86 Mahajan D, Liang JJ, Tsou CH. Toward understanding clinical context of medication change events in clinical narratives. AMIA Annu Symp Proc 2022; 2021: 833-842
• 87 Fong A, Kim TC, Ratwani RM, Kellogg KM. Task2Heart: exploring heart rate differences with time-motion workflow observations of emergency medicine physicians. J Med Syst 2018; 42 (09) 170
• 88 Ratwani RM, Savage E, Will A. et al. A usability and safety analysis of electronic health records: a multi-center study. J Am Med Inform Assoc 2018; 25 (09) 1197-1201