Optimization of Data Collection during Public Health Emergencies—Experience with APACHE II Score

Abstract Background Capturing accurate clinical data in real time is a challenge during public health emergencies. The United States Critical Illness and Injury Trials Group-Program for Emergency Preparedness is committed to improving these preparedness efforts. Objectives We aimed to create an electronic Acute Physiology and Chronic Health Evaluation (APACHE) II data collection instrument that (1) leverages Research Electronic Data Capture (REDCap) automated calculations and logic, (2) may be shared across sites, (3) overcomes limitations in existing APACHE II instruments in the REDCap library, and (4) suggests changes to be made to data collection instruments during emergencies. Methods The APACHE II instrument was implemented using REDCap. Data fields were divided into four sections: age, Acute Physiology, Glasgow Coma Scale, and chronic health status. Usability testing was followed by two preliminary evaluations: a comparison to existing APACHE II instruments and a simulated emergency exercise. Results The final instrument consisted of 34 data fields. It produced an accurate APACHE II score and was faster to complete than two previous implementations (average of 97.5 seconds vs. 323.5 and 183.5 seconds). During the simulated emergency exercise, the instrument was used at 10 sites to create 34 patient records; median time to complete the instrument was 150.5 seconds. Conclusion This project demonstrated feasibility of improving the accuracy and efficiency of a data collection instrument. Future efforts should focus on expanding these methods to develop other scoring tools for use during emergencies and additional testing to ensure it is ready for use during a real emergency.

priorities and prepare forms to capture data. Having these forms prespecified ensures that they can be deployed in a timely manner during a real emergency. Validated scoring tools are one type of data capture form that is important to have ready for rapid deployment. These tools facilitate information sharing between clinical sites, comparisons across studies, and reproducibility of methods and analyses. 2 Similarly, the use of widely available technologies and software to collect such data is important to promote participation by as many sites as possible. 3 The United States Critical Illness and Injury Trials Group-Program for Emergency Preparedness (USCIIT-PREP) is a group of clinicians who are committed to building the capacity for such research and preparedness efforts. 4 USCIIT-PREP's focus is on developing communication, data collection, and data reporting networks that would allow sites to rapidly deploy clinical research protocols in the event of a public health emergency. 5 The specific threats that USCIIT-PREP's work addresses include infectious diseases that lead to critical illness such as influenza, anthrax, and severe acute respiratory infection.

REDCap
Research Electronic Data Capture (REDCap) is an electronic data collection tool, created by Vanderbilt University and a consortium of institutions, which is widely used in academic medical centers for clinical trials and other research studies. 3 Main advantages of REDCap are that it is user-friendly, intuitive, and allows custom calculations and logic to be incorporated into instruments. Another advantage is that REDCap is available at no cost for consortium participants. The literature has demonstrated REDCap's flexibility and customizability by using it to build data management systems for a variety of purposes and subject areas, including ARDS/sepsis patient data, 6 genitourinary oncological data, 7 and pacemaker longterm outcomes. 8 REDCap is an effective tool for collecting data across multisite studies, which is especially important for gathering sufficient data to investigate rare conditions. 9 Another useful feature of REDCap is the Shared Data Instrument Library, which allows researchers across the REDCap consortium to share and reuse instruments. This resource aims to reduce the amount of time spent building data collection instruments by decreasing duplication of effort in the design phase, thereby allowing instruments to move from development to production more quickly. The Shared Library also promotes the use of validated tools and data standards to facilitate data sharing across institutions and studies. 10 The ability to quickly deploy shared data collection instruments is especially important due to the time-sensitive nature of many public health emergencies. 11

APACHE II
The Acute Physiology and Chronic Health Evaluation (APACHE) II score is used in the critical care setting to evaluate patients' severity of disease and risk of death, as well as other adverse outcomes. 12 The APACHE II score can also be used to stratify patients by risk level, aiding in the evaluation of clinical therapies and interventions. 12 APACHE II has been indepen-dently validated for use in several acutely ill patient populations, such as breast cancer patients in the intensive care unit (ICU), 13 neurological ICU patients at risk of nosocomial infection, 14 and hemorrhagic stroke patients. 15 The APACHE II scoring tool provides a quantitative assessment of a patient's mortality risk by assigning points in three categories: the patient's age, the patient's Acute Physiology measurements, and the patient's chronic health status (►Table 1). In each category, the possible values for each measurement are separated into ranges and each range is assigned a point value. Normal measurements are assigned zero points, and increasingly abnormal measurements receive higher points. For example, age points increase with the patient's age, whereas chronic health status points are assigned based on a patient's health history and admission status. Nonoperative and emergency surgery ICU patients receive a higher number of chronic health points than patients admitted for elective surgery. The addition of all points in an individual category constitutes the category "subscore." The sum of the age, acute physiologic measurements, and chronic health status subscores equals the total APACHE II score, which can range from zero (best) to 71 (worst). 12

Current APACHE II Implementations
Two APACHE II implementations currently exist in the REDCap Shared Library. 10 However, these forms have several limitations in the context of an emergent situation, where rapid and accurate data collection is critically important. The first implementation, Version 1, provides space for point values to be entered manually for age, physiological measurements, and chronic health, but does not provide guidance on what those point values should be. The second implementation, Version 2 (with scoring), automatically calculates subscores and the final APACHE II score, but it requires all data to be entered twice: once in a free text field for the specific value, and once as a multiple-choice question to assign a point value to the measurement. Addressing these issues-uncertain instructions for assigning points and the time burden posed by double data entry-is important to make the APACHE II instrument appropriate for use in an emergency situation.

Objectives
The objective of this article is to describe the methodology used to create an electronic APACHE II instrument that (1) leverages REDCap functionality for automated calculations and logic, (2) may be shared across research sites using REDCap's shared library, (3) overcomes limitations in existing shared instruments for APACHE II in the REDCap library, and (4) provides a paradigm for data collection instruments that can be used during public health emergencies.

Methods
The APACHE II form was implemented in REDCap version 6.5.8. As the data collection instrument created in this study was intended to be used as an electronic Case Report Form (eCRF), best practices for designing CRFs and/or eCRFs were ACI Open Vol. 3 No. 1/2019 considered throughout the design process. These guidelines indicated that many of the challenges in design, collection of precise data, and user-friendliness could be addressed during the creation process with careful planning by a multidisciplinary team. 16 To that end, our team included informatics, critical care, nursing, software development, and data experts. Additionally, our process included several steps to intentionally elicit input from additional subject matter experts during the initial implementation and evaluation. Best practices also guided the design of the instrument itself. 16 Among the recommendations incorporated in our implementation of the USCIIT-PREP APACHE II instrument were consistent formatting, clear and concise questions, clear skip patterns, avoidance of "check all that apply," specification of units, and use of consistent data formats. These details are described in the following subsections.
To facilitate data entry, we divided the three APACHE II original categories into four sections: age, Acute Physiology, Glasgow Coma Scale (GCS), and chronic health status. The original APACHE II instrument included GCS as one of the 12 measurements in the Acute Physiology category. However, to provide complete information to clinicians and accurately determine the number of points assigned to a patient's GCS, our APACHE II instrument placed GCS in a separate section with its own subscore (see ►Table 1).

Age
This section was configured as two data fields: a multiplechoice question where the user could select the appropriate age from a list, and a calculated field that generated the age subscore. In the first field, each answer choice contained an age range that corresponded to 0, 2, 3, 5, or 6 points, with increased age corresponding to a higher score (as shown in ►Table 1). The choice to report age using a multiple-choice field, rather than free text, reflected best practices for CRF design by minimizing the chance of a typing error. The  subscore was configured as a calculated field whose value equaled the points assigned to the age chosen in the preceding question.

Acute Physiology
The Acute Physiology subscore of our APACHE II instrument consisted of 11 measurements, (►Table 1). Measurements were configured as multiple-choice fields where only one answer choice could be selected. Each answer choice contained a range of physiological values and each range was assigned a point value: zero for the range containing the normal measurement, and increasing point values for increasingly abnormal ranges. To facilitate data entry, the ranges were reorganized so each numerical range was presented as its own answer choice, even if more than one range was assigned the same point value, and ranges were presented in ascending numerical order for clarity. ►Table 2 shows an example of the reorganization of the answers made to improve clarity.
In the original APACHE II, data must be entered for three pairs of questions. The first of these pairs is the partial pressure of O 2 in arterial blood (PaO 2 ) or the alveolar to arterial (A-a) Difference. Whether the clinician should enter PaO2 or A-a difference depends on the patient's fraction of inspired oxygen (FiO 2 ) measurement. The second set of paired questions is the arterial pH or serum bicarbonate: two different laboratory values, of which one is entered at the user's discretion. The third pair of questions is the creatinine level, with different response choices depending on whether the patient is in acute renal failure or not. For each of these paired questions, only one of the two measurements should be included in the calculation of Acute Physiology subscore. The USCIIT-PREP APACHE II instrument was built to prevent users from entering incorrect data by hiding each pair from view using branching logic. An "either/or" question was inserted before each pair to specify which measurement would be entered. Upon answering the preceding "either/or" question, the corresponding field would appear for the end-user to answer. For example, in the PaO 2 or A-a Difference pair, the end-user was asked first for the FiO 2 measurement, and, depending on the answer, the corresponding question would be displayed. No points are assigned to the "either/or" questions (e.g., FiO 2 ).
Improvements were also implemented in the Acute Physiology subscore, which was designed as a calculated field. In accordance with best practices for CRF development, the equation for this field was designed to fulfill several require-ments. First, it needed to assign point values to each of the answer choices for every field in this section (since many questions had multiple answer choices that received the same number of points). Second, since REDCap's default functionality is to require all fields included in a calculation to be filled for the calculation to execute, the calculation needed to execute even when some fields in the section remained unfilled. Finally, it needed to add all of the point values for the Acute Physiology section without requiring the user to perform any calculations by hand or reference another source for scoring guidelines.

Glasgow Coma Scale
The original APACHE II form included points for a patient's GCS as a component of the Acute Physiology score. However, it simply asked for the "Glasgow Coma Score points," with a note saying to calculate it as "15 minus the actual GCS," without providing any guidance on how to directly calculate "actual GCS." To facilitate data entry, we chose to include questions to help end-users determine a patient's actual GCS.
The GCS consists of three questions regarding a patient's eye-opening response, motor response, and verbal response. These three questions were configured in REDCap as radio button fields, with a point value assigned for each response (see ►Table 1 for point values). Two calculated fields, "Glasgow Coma Scale Total" and the Glasgow Coma Score points ("GCS Subscore") were included in this section. GCS Total was automatically calculated as a sum of the point values for eye, motor, and verbal responses, and the GCS subscore was automatically calculated as 15 minus actual GCS. Calculated fields and radio buttons, rather than free text, were used to promote data consistency and reduce error. For clarity, these questions were separated into their own section rather than being included in the Acute Physiology section, as shown in ►Fig. 1.

Chronic Health Status
Points are added to the APACHE II score based on the purpose of a patient's hospital admission only if the patient had prior chronic health problems. In the original APACHE II form (and in previous REDCap implementations), the criteria to determine if a patient has chronic health problems were grouped together in one long paragraph. For clarity, we separated the individual conditions from this paragraph into a series of 5 "yes" or "no" screening questions. Answering yes to any one of these questions indicated the presence of prior chronic illness and would trigger a question to appear regarding the patient's hospital admission status. Two points were assigned for the option "Admission for elective operation," while five points are assigned for the answer "Non-surgical admission or admission for emergent operation," as shown in ►Table 1.
The chronic health status subscore was configured as a calculated field, where the value reflected the points for the patient's hospital admission status, or zero points if all five of the chronic health screening questions were answered "no."

Final APACHE II Score Calculation
The APACHE II score was given by a calculated field where the equation is the sum of the age, Acute Physiology, GCS, and

Preliminary Evaluations
Usability testing and preliminary evaluation of the instrument were conducted using data derived from four published case reports of critically ill influenza and pneumonia patients. [17][18][19][20] Published case data of critically ill respiratory patients was used because of its availability, suitability for preliminary data entry evaluation, and applicability to USCIIT-PREP's mission of conducting research among critically ill patients. The USCIIT-PREP APACHE II instrument was first validated by three informatics experts, with special attention to usability and data consistency. Phase one of preliminary evaluation compared the USCIIT-PREP APACHE II Instrument to the two APACHE II instruments previously available in the REDCap Shared Library. Several characteristics of the instruments were compared, including the total number of fields, field types, number of manual versus automatic calculations, and the minimum number of clicks required to complete each instrument. Finally, a second phase of preliminary evaluation of the APACHE II instrument was conducted during a USCIIT-PREP preparedness study of critically ill patients with influenza. This study included clinical sites across the country and was intended to test a larger data collection instrument, which included APACHE II, more thoroughly. The test data from the published case reports were used during the preparedness simulation, along with additional details provided by clinicians at the study sites. In both phases of evaluation, the time taken to complete each instrument was tracked automatically through REDCap logs and calculated as the aggregate time spent on each form (page), inclusive of all sessions when the same form was saved and returned to later. The implementation process in REDCap including initial configuration, testing, revisions, and final adjustments took 40 hours total.

Results
The four sections of the final USCIIT-PREP APACHE II instrument consist of a total of 34 data fields, 21 of which are marked as required. Twenty-seven (79.4%) are radio button fields where the clinician must select only one answer, and seven (20.6%) are automatically populating fields, including six calculations and the participant identification number assigned by REDCap. There are no free text fields in the instrument. The six calculated fields contain variables that reference previous fields in the instrument. The number of variables per calculation ranged from 1 to 17, with a median of 3 (►Table 3). Although each patient's Acute Physiology subscore is the sum of 11 measurements, the calculation incorporated 17 variables to account for the different possible combinations of choices in the three sets of paired questions.
During usability testing by informatics experts, a total of three patient records were entered into the APACHE II instrument. These three records contained data in 17 (81%), 8 (38%), and 1 (5%) of the 21 required fields. Minor issues were identified by the informatics experts and subsequently resolved by consensus. For example, in the chronic health section, the original answer choices "1) Non-surgical or emergent operation; or 2)  Elective operation" were determined to be ambiguous. At first glance, it was unclear to which category medical patients would belong because both answer choices appeared to refer only to surgical patients. Therefore, the wording of the two choices was changed to "1) Non-surgical admission or admission for emergent operation; or 2) Admission for elective operation" to distinguish nonsurgical patients from other types of patients. Another change made for data consistency was the addition of the brief instruction "Use worst values within the first 24 hours of ICU admission" at the top of the form. This note was included to prevent incorrect data being entered in cases where more than one value or measurement might be available.
The evaluation comparing the USCIIT-PREP APACHE II instrument to the two existing REDCap instruments was done by the first author and is summarized in ►Table 4. This comparison confirmed that the final APACHE II score was the same for each patient across all three instruments. However, while the final score was the same, the time needed to complete each instrument was substantially different. Versions 1 and 2 contained more free text fields and manual calculations than the USCIIT-PREP APACHE II instrument, which in turn contained more automatically calculated fields and more multiple-choice fields (►Table 4). Each multiplechoice field was completed with one click. While completing Version 1, both APACHE II and GCS references had to be consulted to determine various point values, contributing to the overall time. While completing Version 2, the GCS reference was needed. The USCIIT-PREP APACHE II instrument did not require any external resources and had the lowest average completion time.
During the USCIIT-PREP public health emergency preparedness study, 34 simulated cases were created by 10 sites across the U.S. End-users who completed these cases were clinical research staff, including physicians and nurses, with a range of prior REDCap experience. The average time spent completing the APACHE II instrument was 257.06 seconds, or 4.3 minutes (minimum 22 seconds, maximum 954 seconds, median of 150.5 seconds). The 21 required fields ranged in completeness from 35.3 to 97.1% in the 34 patient records. Fields completed in only 35.3% of patient records include serum sodium, serum potassium, hematocrit, white blood cell count, and GCS eye, verbal, and motor assessments. Age was complete in 97.1% of records.

Discussion
The ability to conduct research during public health emergencies is vital to enhance ongoing responses and to prepare for future events. 11 Considering the many challenges faced by clinicians and other responders when treating patients during emergencies, it is important for research instruments to be designed in a way that allows data to be collected quickly and accurately without interfering with treatment. 5 This study demonstrates the ability to leverage REDCap to design a data collection instrument that can achieve these aims.
The USCIIT-PREP APACHE II instrument represents several improvements over previous implementations in the context of public health emergencies, when time, supplies, and medical staff are under stress. For example, the Glasgow Coma Score points (GCS subscore) form is designed to save time and improve data consistency by minimizing the effort required and using automatically calculated fields. Providing sufficient detail and assistance to end-users is especially important because the reliability of GCS has been found to depend on the experience and training of the scorer, as well as the consistency of the scoring instrument. 21 Data collection instruments that maximize consistency are especially important during an emergency, when end-users may include individuals with less training or experience completing specific scoring tools.
Another change that aims to facilitate optimal medical care during an emergency is the reorganization of answer choices (as in ►Table 2) for data fields in the Acute Physiology section, providing a more user-friendly layout. In the previously available Version 2 of the APACHE II instrument, the multiple-choice questions were organized so that each answer choice included the range or ranges corresponding to one particular point value. For example, one point was given for a temperature measurement of "34-35.9 or 38.5-38.9" degrees Celsius. Including nonconsecutive ranges in one answer choice, separated by the word "or," could cause confusion and delays in the data entry process. Instead, the USCIIT-PREP APACHE II instrument lists all numerical ranges in ascending numerical order. This rearrangement necessitates a more complex equation for the calculation of the Acute Physiology subscore than the previous arrangement does. However, considering the context of emergency preparedness, the choice to use a more complex calculation results in a more user-friendly and efficient data entry format for clinicians.
Finally, the assignment of point values to measurements in the Acute Physiology section improves upon previous implementations by completely automating the process. Version 1 of the instrument does not provide any point values or calculations, requiring clinicians to consult an external APACHE II source to determine point values for each measurement and add up scores by hand. Version 2 of the instrument does include point values, but requires each data field to be entered twice (once as text and once as a radio button) and utilizes the less intuitive multiple-choice answer format described above. The USCIIT-PREP APACHE II instrument, on the other hand, includes calculated fields to both assign point values and automatically add up scores with just a single click for each measurement. Preliminary evaluation of the USCIIT-PREP APACHE II instrument confirmed that it is faster to complete than previous implementations, while still providing an accurate final score. Recommendations from expert reviewers obtained during usability testing confirmed the importance of carefully considering wording of questions and instructions, helping end-users understand exactly what data to enter. This is especially important for data consistency and for the validity of the final APACHE II score.
The comparison of the three APACHE II instruments highlighted the differences in the design and the effort required to complete the three instruments. In general, having fewer free text fields and manual calculations provides less opportunity for error, while having more automatic calculations and multiple-choice questions improves both efficiency and accuracy. The simulated public health emergency exercise resulted in a slightly higher median time to complete the instrument than the instrument comparison (150.5 vs. 97.5 seconds). However, this difference may be due to the fact that many clinicians involved in the emergency simulation had to supplement test cases with additional data and were also using the complete USCIIT-PREP data collection instrument for the first time. In addition, the participation of multiple sites provided the opportunity for a wider group of users, most of whom were familiar with REDCap, to use the instrument and provide feedback. Anecdotally, a few users commented that they found the new USCIIT-PREP APACHE II instrument better and easier to use than either of the previous APACHE II instruments in the REDCap Shared Library. Although the simulation also resulted in several patient records with missing data, cases entered using medical record data from a small number of real patients (not described in this article) produced complete records, as expected.
Limitations of the USCIIT-PREP APACHE II instrument include that it represented only one scoring tool among many that are commonly used in clinical settings, and it was tested using a small number of cases in a nonemergency situation. More data about the instrument will be available after the upcoming influenza season, when there will be further opportunity for it to be used prospectively with real patients undergoing treatment in the ICUs of multiple hospitals across the country. The instrument has been submitted to the REDCap Shared Library and is pending approval. To continue building the capability for data collection across multiple sites during emergencies, future work should include identifying other validated scoring instruments to develop in a similar manner. Future work should also extend our evaluations by involving a wider range of third-party end-users and by utilizing medical record data.

Conclusion
This study demonstrates improvements to a data collection instrument that result in improved timeliness and accuracy, both of which are critical during the response to a public health emergency. Future work is needed to create a library of readily available data collection instruments that can be deployed in an emergency situation. The ability to collect and share data during public health emergencies depends on the development of this research infrastructure.

Clinical Relevance Statement
This project demonstrates the feasibility of building a data collection instrument that can capture accurate data in a short amount of time. These methods can be used to prepare for clinical research efforts during public health emergencies.

Protection of Human and Animal Subjects
This study did not involve any human/animal subjects.

Funding
This work was partially funded by a contract from the US Food and Drug Administration.