Development, Implementation, and User Evaluation of COVID-19 Dashboard in a Third-Level Hospital in Iran

Abstract

Introduction The implementation of a dashboard enables managers to make informed and evidence-based decisions through data visualization and graphical presentation of information. This study aimed to design and implement a COVID-19 management dashboard in a third-level hospital in Mashhad, Iran.

Materials and Methods This descriptive developmental applied study was conducted in the second half of 2020 in three stages, using user-centered design methodology in four phases: (1) specification of the application context, (2) specification of requirements, (3) creation of design solutions, and (4) evaluation of designs. Data collection in each phase was performed through holding group discussions with the main users, nominal group techniques, interviews, and questioners. The dashboard prototype for the data display was designed using the Power BI Desktop software. Subsequently, users' comments were obtained using the focus group method and included in the dashboard.

Results In total, 25 indicators related to input, process, and output areas were identified based on the findings of the first stage. Moreover, eight items were introduced by participants as dashboard requirements. The dashboard was developed based on users' feedback and suggestions, such as the use of colors, reception of periodic and specific reports based on key performance indicators, and rearrangement of the components visible on the page. The result of the user satisfaction survey indicated their satisfaction with the developed dashboard.

Conclusion The selection of proper criteria for the implementation of an effective dashboard is critical for the health care organization since they are designed with a high-tech and content-based environment. The dashboard in the present study was a successful combination of clinical and managerial indicators. Future studies should focus on the design and development of dashboards, as well as benchmarking by using data from several hospitals.

 

MissingLinkText

Keywords

COVID-19 - dashboard - hospitals - information systems - Iran

Background and Significance

The coronavirus disease 2019 (COVID-19) was first detected in Wuhan, China, on December 30, 2019. The World Health Organization recognized the disease as a global epidemic on March 11, 2020.[1] The outbreak quickly turned into a critical condition for the health systems around the world.[2] [3] Given the priority of treating COVID-19 patients, health care systems devoted the majority of their resources to combat the epidemic, which in turn led to the reorganization of health care services.[2] [4] Decision making during the COVID-19 pandemic was a challenge for health system managers, especially hospital managers at all organizational levels, due to the unknown characteristics of the emerging disease; lack of vaccines, drugs, and acceptable therapies; the complexity of the disease in terms of clinical manifestations and modes of transmission; and the unpredictable consequences of the disease on one hand and its devastating and deadly waves on the other.[5] Moreover, the decisions of health system managers at different levels have an important impact on the effectiveness of health services[6] and the success of health organizations in controlling the COVID-19 pandemic.[4]

In the case of the COVID-19 pandemic, the reaction of health systems has been faster and much more complex than normal.[7] However, health system managers and decision makers faced more challenges in the process of decision making and the provision of appropriate and timely responses due to the lack of proper understanding of the environment and the high volume of information they are confronted with in critical situations.[8] [9]

Based on the previous experiences, the main challenges of decision making in the current situation include misunderstanding the scope of the current situation, as well as information overload experienced by the decision makers.[8] In such critical situations, it is very important to design a suitable system that collects, analyzes, and reports information from available sources.[10] In addition, having access to high-precision reports can facilitate the implementation of support operations, identification of system weaknesses, and provision of effective solutions for reliable planning.[11] [12]

In this regard, technical advancements and integration of all provided information in a suitable field accelerate decision making.[13] [14] Data visualization has provided a huge capacity for the examination of different dimensions of information and the visual exploration of the relationships among different components.[15] Dashboards are one of the most important data visualization tools.[16] They make data evaluation easier and assist users in analyzing information through the processing of information and identification of factors that can be evaluated.[17] Consequently, dashboards provide the possibility of informed and evidence-based decision making for the managers.[15]

The first step in the development of a dashboard is the selection of main performance indicators and the determination of a basic relationship between them.[18] Research showed that a dashboard should provide users with content that is tailored to their needs and be designed in a way to be easily understood by different users.[19] [20] It should be noted that valid and reliable data for the user should be displayed completely, accurately, and timely.[20] Moreover, the dashboard should be easy to use and facilitate the process of decision making.[21]

The challenges of patient care administration and monitoring suspect cases made it vital to know about the number of daily cases, diagnosis results, and transfers of COVID-19 patients to hospitals for treatment. During the COVID-19 epidemic, various studies have designed information dashboards at different levels to better manage COVID-19 patients.[22]

In most previous related studies, information dashboards have been designed nationally to address the demographic characteristics of patients with COVID-19. Berry et al developed open access epidemiologic data and an interactive dashboard to monitor the COVID-19 outbreak in Canada, which included such information as demographic characteristics, location, report date, travel history, and exposure source. All data are openly accessible and updated daily.[23]

Dong et all developed an online interactive dashboard, hosted by the Center for Systems Science and Engineering at Johns Hopkins University, Baltimore, Maryland, United States, to visualize and track reported cases of COVID-19 in real time.[24]

Moreover, some studies have designed hospital-level dashboards to track the status of COVID-19 patients in hospital wards. For example, Vizcaychipi et al developed a COVID-19 near real-time traffic light system in an acute hospital setting. The input variables were age, gender, first recorded blood pressure, respiratory rate, temperature, heart rate, indices of oxygenation, and C-reactive protein.[25]

Therefore, this study aimed to design (develop) and implement a COVID-19 management dashboard at the hospital level in Iran.

 

MissingLinkText

Methods

This descriptive developmental applied study was conducted to develop and assess a COVID-19 dashboard in a big hospital in Mashhad, Iran, using a focus group technique and a panel of experts. This dashboard was designed in Imam Reza Hospital, which is a tertiary care teaching hospital in Mashhad, Iran. This hospital has approximately 1,000 beds and provides care to around 6,000 inpatients and 19,000 emergency patients monthly.

User-centered design (UCD) was adopted for the development of the dashboard. The UCD is an iterative design process in which designers and other stakeholders focus on the users and their needs in each phase of the design process.[26] The UCD process ensures a stable integration of later users throughout all phases of the project. With more user focus, the UCD method is one of the most suitable methods for the recognition of the user's problems.[26] [27]

The general phases of the UCD process include:

1. Specification of the context of use: Identification of the people who are supposed to use the product, the purpose of use, and the conditions of use.

2. Specification of requirements: Identification of any business requirements or user goals that must be met for the success of the product.

3. Creation of design solutions: This part of the process may be done in stages, building from a rough concept to a complete design.

4. Evaluation of designs: Evaluation—ideally through usability testing with actual users—is an integral part of software development.[26]

In other words, UCD steps for the development of a dashboard include identification of goals, users, data, design and layout, user feedback, dashboard design, and overall best practices.

This study was performed in four steps according to UCD stages.

The main goals of designing this dashboard for real-time analysis and management of COVID-19 disease and the general features of this dashboard were examined through brainstorming during the first focus group discussion. This focus group discussion was held with the presence of 10 potential users of the system, including hospital managers, nursing managers, heads of relevant departments (infectious disease specialists), and heads of COVID-19 committees (emergency medicine specialists). The main criteria for the selection of participants included their experience and knowledge of hospital management and their willingness to participate in the study.

In the second phase, the focus group discussion was held in the hospital, with those who were present in the previous focus group, and the meetings were chaired by the hospital manager. Initially, users were asked such questions as “What information do they need to better manage COVID-19 in their field?,” “Why do they need this information?,” “When do they need this information?,” “How should they access this information?,” and “Where should they get the needed information?.”

Moreover, the focus group discussed any user goals that must be met for the success of the product. Therefore, the necessary approach for getting access to each piece of information, as well as the information sources for inclusion in the COVID-19 information management dashboard was identified subsequently.

At the end of this phase, the comments were written and summarized, and voting was done using the nominal group technique in the form of a list of information that needs to be placed in the COVID-19 information management dashboard. The nominal group method was used to analyze the focus group data and get a conclusion at the end of the sessions. Access to each required indicator and information resource was provided and approved by the majority (75%) of the participants.

Subsequently, the study team including five specialists (i.e., two physicians, one nurse, one health information management, and one manager) classified indicators in three classifications of inputs, process, and output after the second focus group meeting, according to related previous studies.

In the creation of the design solutions phase, the specialized software design team developed the dashboard prototype using the Power BI Desktop software based on the identified data elements and requirements from the previous step.

The use of colors in a dashboard can be a plus; however, it is necessary to choose them according to known rules, as indicated in the study conducted by Pestana et al.[28] In this regard, different colors were applied to indicate the importance of information. In this regard, dark-colored fonts were used to highlight information that required more user's attention,[28] and the barely discernable pale background color was used to provide a more soothing and less starkly contrasting surface on which the data can reside.[29]

Afterward, the designed prototype was integrated with the hospital information system (HIS) and its capabilities and visual features were displayed to the users. In the final phase (i.e., evaluation of the designs), 1 week after the prototype was available to users, the third focus group was held with the same participants who had attended the second focus group. These 10 main users were asked to comment on suggestions provided for upgrading the designed dashboard. Furthermore, their satisfaction was measured based on the end-user satisfaction model with such items as content review, accuracy, ease of use, and timeliness using a questionnaire that was scored based on a five-item Likert scale, similar to that used in the Rouhani and Zamenian study,[21] in which very good = 4, good = 3, moderate = 2, bad = 1, and very bad = 0. The validity and reliability of the questionnaire were confirmed. The dashboard software was then provided to participants to evaluate after working with the dashboard through the completion of a designed paper questionnaire. The mean ± standard deviation (SD) of the questions were calculated as well.

Eventually, the final version of the dashboard system was developed after the comments and suggestions received from users and the required changes were applied to the designed prototype.

 

MissingLinkText

Results

Opinions from potential users of the COVID-19 information management dashboard were gathered in the initial phase of the study. According to the comments, objectives of dashboard development included the possibility of informed and evidence-based decision making for the managers, easier analysis of information through information processing, and identification of the main performance indicators and factors that can be evaluated. Regarding general features dashboards, participants agreed that dashboards are likely to solve the problems of presentation format and information load when certain visualization principles and features are present (e.g., high data-ink ratio and drill-down features). Dashboards should have some level of flexibility so that users can switch between alternative presentation formats. Pop-ups and warnings can help users select an appropriate presentation format.

At the end of the session, the main users of the dashboard were given their votes. The criteria for selecting these individuals included membership in the hospital board of directors or one of the COVID-19 committees, such as the Nosocomial Infection Committee, and at least 2 years of working experience in the hospital.

The mean ± SD age and mean ± SD work experience of the selected main users were 44 ± 4.11 and 12 ± 6.04 years, respectively. The participants in the study included four females and six males and their demographic information is presented in [Table 1].

Table 1 Demographic characteristics of participants

Variable	Frequency (%)
Gender
Female	4 (40)
Male	6 (60)
Age[a]
36–41	3 (30)
42–47	5 (50)
48–53	2 (20)
Education status
Bachelor of Science	4 (40)
Master of Science	3 (30)
Specialist	2 (20)
PhD	1 (10)
Field of study
Health economics	1 (10)
Health information management	2 (20)
Management	2 (20)
Nursing	3 (30)
Medicine	2 (20)
Work experience[b]
5–10	4 (40)
11–16	2 (20)
17–22	3 (30)
23–27	1 (10)

^a Mean ± standard deviation (SD) age is 44 ± 4.11.

^b Mean ± standard deviation (SD) work experience is 12 ± 6.04.

In the second phase, the main information requirements of caring for COVID-19 patients were identified with the participation of the main users. Based on the available data, 25 indicators have been selected to be displayed in the COVID-19 dashboard. For a better understanding of information, participants decided to classify the selected indicators into three categories (performance, clinical, and demographic categories) ([Table 2]). Data on these indicators were extracted from the HIS, automatically. Moreover, at the request of participants in the focus group, users could visualize the data summary of each patient and compare the data obtained from different departments. It was suggested that the number of admissions, discharges, and deaths over a period of time could be observed as well. Moreover, it was decided to use a table to show the average request for the diagnostic tests.

Table 2 Indicators for display in a dashboard

Criteria	Indicator	Percentage of agreement
Input	The number of patients admitted to triage	76%
	The number of patients admitted to the emergency department	88%
	The number of patients referred from other centers	78%
	The Number of empty beds	94%
	The gender of patients	96%
	The age of patients classified into 19 groups (based on the standard table of life expectancy pertaining to World Health Organization [57])	81%
	Location of living (Urban or rural)	77%
	Job title (Based on 6 job titles, including unemployed, employed, retired, housekeeper, student, health professional identified in HIS)	96%
	The number of severe COVID-19 patients	92%
Process	The Number of occupied beds	75%
	The mean length of stay in departments for COVID-19 disease	95%
	The mean length of stay in ICUs for COVID-19 disease	95%
	The mean time between graph request and response	99%
	The mean time between laboratory tests request and response	84%
	The mean time between CT-scan request and response	93%
	The number of laboratory tests requests	84%
	The number of requested radiographies	82%
	The number of CT-scan requests	93%
	The number of COVID-19 specific drugs	93%
	The number of patients ready to be discharged from the inpatient department	91%
	The number of patients ready to be discharged from an emergency department	87%
Output	The number of deaths	93%
	The number of patients discharged	84%
	The number of patients referred to other centers	98%
	The number of patients leaving the triage	88%

Abbreviations: CT, computed tomography; HIS, hospital information system; ICU, intensive care unit.

Dashboard users could view demographically classified data, such as gender-based data. In total, eight items were suggested by the second focus group as dashboard requirements: (1) the ability to access the system through Web browsers without the need to install a specific platform or operating dashboard, (2) the ability to connect to the health information system for data gathering and analyzing, (3) the capability to define access levels for the users, (4) the possession of user-friendly dashboard, (5) the ability to change, develop, configure, and service the dashboard, (6) the use of color capabilities in the dashboard, (7) the ability to report online and drill down, and (8) the possibility to visualize data over a period of time.

Subsequent to the development of the initial dashboard prototype using the Power Bi server, each user was given a URL to use the dashboard, and access levels were also defined.

According to the access level set in the initial focus group of the prototype dashboard, it was possible to provide daily, weekly, and monthly reports according to the selected indicators. It was also possible to create exception reports based on the users' requests. This prototype dashboard utilized data from March 2020 to August 2020 based on HIS. The HIS is an administrative and clinical database containing data from all departments in the hospital, which provided care to COVID-19 patients. The prototype dashboard presents summary data of COVID-19 patients at the hospital level for the main users.

In the third step of the focus group, suggestions were made by the members of the focus group to upgrade the dashboard according to the feedback provided by the users. After consultation with the technical team for the modification of the relevant dashboard, it was agreed that the number of deaths, admissions, and discharges should be displayed in three separate graphs, and the distribution of patients by days of hospitalization and the average length of stay be displayed in a graph as well. Afterward, a questionnaire was provided to participants and their satisfaction with the designed dashboard was measured. The internal and external validity, reliability, and statistical validity of this questionnaire have been approved by Aggelidis and Chatzoglou.[30] The average user satisfaction of the model is presented in [Table 3].

Table 3 User satisfaction of the dashboard

Variables based on EUCS model 29	Research variables	Average
The independent variable format	Satisfaction with the dashboard format	4.782 ± 0.321
The independent variable being up to date	Satisfaction with up-to-date status of the dashboard	4.402 ± 0.987
The independent variable ease of use	Satisfaction with the dashboard's ease of use	3.985 ± 0.654
The dependent variable final user satisfaction	Overall satisfaction of the dashboard	4.36 ± 0.451
The independent variable content	Satisfaction with dashboard content	4.46 ± 0.475
The independent variable accuracy	Satisfaction with the accuracy of the dashboard	4.191 ± 0.430

Abbreviation: EUCS, end-user computing satisfaction.

The images of the report page of the designed information dashboard are presented in [Figs. 1] [2] [3].

Fig. 1 The final dashboard visualizes data for patients screened for COVID-19 disease. The prototype dashboard presents a deidentified patient record populated with clinical data (imaging diagnostic tests).

Fig. 2 The number of daily admissions, daily discharges, and daily deaths in a sample time period.

Fig. 3 Distribution of length of stay and mean ± standard deviation (SD) of patient age in deidentified department.

 

MissingLinkText

Discussion

As previously mentioned, the main objective of the present study was to develop and implement a dashboard system to manage the crisis caused by COVID-19 in a third-level hospital. In this regard, the main indicators and requirements of such a system were identified. Therefore, identification of the main indicators and requirements of the developed dashboard through the participation of potential users in all stages of development and implementation was the difference between this study and similar studies.[25] [31]

Some of the previous studies on dashboard design for pandemic diseases, such as COVID-19,[1] [4] severe acute respiratory syndrome, and H1N1 influenza,[32] [33] [34] had their focus on the clinical aspect and analysis of data outcomes. The ignorance of users' views and their participation in designing such information systems has led to their lack of interest in such systems.[35] [36] Therefore, the present study focused on the identification of the information needs and requirements of such a system from the users' point of view. Based on the study results, several methods were used to identify and extract indicators and requirements of the dashboard system which included investigation of similar information systems, user interviews, and questionnaires.[37] [38] [39]

In this study, a UCD was adopted for the development, implementation, and evaluation of the COVID-19 dashboard. The results of a study performed by Setyawan et al indicated that this method had a positive relationship with supporting user's participation in the process of application development.[27] In studies in which a similar method has been used, researchers designed a safety dashboard for patients as well. Their findings showed that this method increased patients' participation.[40] [41] Another study based on this method showed that according to the users' feedback, visualizations improved situational awareness and have probably provided valuable information to facilitate informed operational decision making.[42]

In this regard, indicators and requirements of the dashboard system were identified through the examination of similar dashboard systems and interview with users through focus group meetings.

Users tend to utilize a variety of tools, such as mobile devices, especially smartphones, as well as other electronic devices, including personal computers (PCs), laptops, and tablets to access information in Web-based applications, such as dashboards.[37] [43] The designed dashboard in the present study is Web-based and can be accessed by all users through common browsers on electronic devices, such as PCs, laptops, and tablets, without the need for installation on a specific operating system.

Reasonable decision making is a key element in health care organizations at managerial levels. A dashboard is a tool that can integrate data from several different sources, summarize the key performance indicators (KPIs), and provide them to users in a colorful graphic in real-time to facilitate informed decision making.[38] Studies on the design of COVID-19 management dashboards showed that these dashboards connect to and integrate with other health information systems, such as electronic health records and health information systems, to receive data from these systems.[14] [39] [44] [45] Similarly, the dashboard designed in the present study was integrated with the HIS. Therefore, the data could be used in real-time to display and manage COVID-19 disease.

Determination of accurate KPIs is one of the primary requirements in the development of a dashboard. These indicators should be developed in accordance with the needs and goals of the given organization to provide valid and reliable measurements.[46] [47] [48] Demand for intensive care unit (ICU) services will increase under conditions similar to the COVID-19 pandemic. Therefore, patient triage is critical in situations like this since there is always a risk that patients may not be properly admitted to the ICU which may cause more harm and injury to the patients.[35] Regarding the fact that the triage, emergency, and ICU departments are involved in the process of providing health care services to COVID-19 patients, it is important to monitor the workflow of these units. Accordingly, the indicators required for COVID-19 management in the present study were identified through the assessment of the information needs of dashboard users. These indicators were classified into three main groups based on clinical and managerial needs of users, including performance, clinical, and demographic criteria. Subsequently, the output data and reports of the dashboard were displayed to managers and clinical specialists in form of color charts and tables. Based on the results of previous studies, the use of colors in dashboard design led to a better understanding of the situation and improved data visualization which in turn reduced the response time and error rate.[36]

Similarly, in the study conducted by Grange et al, the related key indicators were identified and applied to the designed COVID-19 management dashboard.[14] Furthermore, interactive Web-based dashboards have been designed during the COVID-19 pandemic surge in different countries to help managers and clinicians better manage the limitations in human resources, medicine, and medical equipment and take more effective actions.[1] [31] [49] [50]

In addition, user satisfaction is one of the important factors affecting the acceptance and application of different software.[51] However, the review of studies related to the design of COVID-19 dashboards showed that user's satisfaction with the designed dashboard has been examined in a small number of them.[52] In the same line, user's satisfaction with the designed dashboard was examined in the present study in terms of the simplicity of use and the extent to which the information and operational needs of dashboard users were met.

As a result, the examination of users' satisfaction and application of their comments and feedback in dashboard design led to improved dashboard performance and increased users' acceptance of the designed dashboard.

It seems that the structure and minimum data included in this dashboard can be used for similar hospitals due to the similar nature of the disease.

Regarding the limitations of the present study, one can refer to the fact that due to the critical situation of the COVID-19 pandemic, focus group interviews with potential users were withheld. Moreover, since the present study was conducted in only one center, the number of participants involved in the development and implementation of the dashboard was small.

 

MissingLinkText

Conclusion

Information dashboards are important for the provision of access to necessary information in the shortest possible time. Information dashboards keep managers informed about the process of patient care, admission, discharge, and incidents, such as the number of deaths and the use of hospital services. The application of this tool leads to more evidence-based decision making in the crisis and reduces the likelihood of error and failure in crisis management.

Furthermore, the simplicity of design and the possibility of connection to the HIS database allows administrators and users to access the reports they need with higher confidence and speed and avoid the repeated collection of the required information.

Eventually, the dashboard developers need to receive periodic feedback from the users, update their information needs, and evaluate the quality of data and dashboard information to preserve the users' desire to use this system continuously and make the most of such information management dashboards.

 

MissingLinkText

Clinical Relevance Statement

This study can be used as a guide for the implementation and development of a dashboard for managing various crises, such as COVID-19 disease in hospitals.

 

MissingLinkText

Multiple Choice Questions

1. According to the USD method, which is not one of the general phases?

   • Specification of the context of the use

   • Specification of requirements

   • Creation of design solutions

   • Development a prototype

   Correct Answer: Option a is the correct answer. The final phase of the USD method is evaluating designs: Evaluation—ideally through usability testing with actual users—is as integral as quality testing is to good software development.

2. Dashboard indicators in this study are divided into how many main categories?

   • One

   • Two

   • Three

   • Four

   Correct Answer: Option c is the correct answer; three categories including inputs, process, outputs.

 

MissingLinkText

 

MissingLinkText

Conflict of Interest

None declared.

Acknowledgments

The authors would like to thank all the staff and managers of Imam Reza Hospital in Mashhad, Iran, for their help and assistance in this project. We also thank Mashhad University of Medical Sciences for supporting this project.

Protection of Human and Animal Subjects

The study was conducted in accordance with the Helsinki Declaration.

Note

This research is part of a large project (Grant Code No. 990315). The study protocol was approved by the Research Ethics Committee of Tehran University of Medical Sciences, Tehran, Iran (Ethical code: IR.TUMS.NIHR.REC.1400.012).

• References

• 1 Florez H, Singh S. Online dashboard and data analysis approach for assessing COVID-19 case and death data. F1000 Res 2020; 9: 570
• 2 Viggiano D, Lococo F, Dell'Amore A. et al; Italian Thoracic Surgery Society (SICT) Task Force on COVID-19 Pandemic. Hospital organization and thoracic oncological patient management during the coronavirus disease-2019 outbreak: a brief report from a nationwide survey of the Italian Society of Thoracic Surgery. Interact Cardiovasc Thorac Surg 2020; 31 (06) 895-899
• 3 Knighton AJ, Ranade-Kharkar P, Brunisholz KD. et al. Rapid implementation of a complex, multimodal technology response to COVID-19 at an integrated community-based health care system. Appl Clin Inform 2020; 11 (05) 825-838
• 4 Yousef M, Fazaeli S, Masoudi S, Masoudi S. et al. Comparative study of the physician specialty and laboratory services requested for patients with COVID-19: evidence from a large hospital. J Isfahan Med Sch 2021; 39 (610) 33-40
• 5 Goniewicz K, Khorram-Manesh A, Hertelendy A, Goniewicz M, Naylor K, Burkle F. Current response and management decisions of the European Union to the COVID-19 outbreak: a review. Sustainability 2020; 12 (09) 3838
• 6 Yousefi M, Assari Arani A, Sahabi B, Kazemnejad A, Fazaeli S. Household health costs: direct, indirect and intangible. Iran J Public Health 2014; 43 (02) 202-209
• 7 Fakhruddin BS, Blanchard K, Ragupathy D. Are we there yet? The transition from response to recovery for the COVID-19 pandemic. Prog Disaster Sci 2020; 7: 100102
• 8 Kitajima M, Ahmed W, Bibby K. et al. SARS-CoV-2 in wastewater: State of the knowledge and research needs. Sci Total Environ 2020;
• 9 Sullivan C, Wong I, Adams E. et al. Moving faster than the COVID-19 pandemic: the rapid, digital transformation of a public health system. Appl Clin Inform 2021; 12 (02) 229-236
• 10 Ye J. Health information system's responses to COVID-19 pandemic in China: a national cross-sectional study. Appl Clin Inform 2021; 12 (02) 399-406
• 11 Lawrence J, Hossain N, Rinaudo C, Buchanan R, Jaradat R. eds. An approach to improve hurricane disaster logistics using system dynamics and information systems. 18th Annual Conference on Systems Engineering Research (CSER), CA, March 19, 2020
• 12 Checchi F, Warsame A, Treacy-Wong V, Polonsky J, van Ommeren M, Prudhon C. Public health information in crisis-affected populations: a review of methods and their use for advocacy and action. Lancet 2017; 390 (10109): 2297-2313
• 13 Ahmadi H, Nilashi M, Ibrahim O. Organizational decision to adopt hospital information system: an empirical investigation in the case of Malaysian public hospitals. Int J Med Inform 2015; 84 (03) 166-188
• 14 Grange ES, Neil EJ, Stoffel M. et al. Responding to COVID-19: the UW medicine information technology services experience. Appl Clin Inform 2020; 11 (02) 265-275
• 15 Joshi A, Amadi C, Katz B, Kulkarni S, Nash D. A human-centered platform for HIV infection reduction in New York: development and usage analysis of the Ending the Epidemic (ETE) Dashboard. JMIR Public Health Surveill 2017; 3 (04) e95
• 16 Fareed N, Swoboda CM, Chen S, Potter E, Wu DTY, Sieck CJ. U.S. COVID-19 State Government Public Dashboards: an expert review. Appl Clin Inform 2021; 12 (02) 208-221
• 17 Buttigieg SC, Pace A, Rathert C. Hospital performance dashboards: a literature review. J Health Organ Manag 2017; 31 (03) 385-406
• 18 Dehghani Mahmodabadi A, Langarizadeh M, Mosaddegh Mehrjardi MH, Emadi S. Development of managerial key performance indicators for a hospital pharmacy digital dashboard. Iran J Pharm Res 2019; 18 (04) 2124-2130
• 19 Hadiwijaya H, Febrianty F, Azdy R. Development of prototype electronic dashboard service performance of LPKA class I Palembang. Sinkron 2021; 5 (02) 394
• 20 Weggelaar-Jansen AMJWM, Broekharst DSE, de Bruijne M. Developing a hospital-wide quality and safety dashboard: a qualitative research study. BMJ Qual Saf 2018; 27 (12) 1000-1007
• 21 Rouhani S, Zamenian SRotbie S. Prototyping and evaluating a hospital dashboard based on end-user satisfaction model. Journal of Health and Biomedical Informatics 2016; 3 (03) 174-185
• 22 Asadzadeh A, Pakkhoo S, Saeidabad MM, Khezri H, Ferdousi R. Information technology in emergency management of COVID-19 outbreak. Inform Med Unlocked 2020; 21: 100475
• 23 Berry I, Soucy JR, Tuite A, Fisman D. COVID-19 Canada Open Data Working Group. Open access epidemiologic data and an interactive dashboard to monitor the COVID-19 outbreak in Canada. CMAJ 2020; 192 (15) E420
• 24 Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis 2020; 20 (05) 533-534
• 25 Vizcaychipi MP, Shovlin CL, McCarthy A. et al; ChelWest COVID-19 Consortium. Development and implementation of a COVID-19 near real-time traffic light system in an acute hospital setting. Emerg Med J 2020; 37 (10) 630-636
• 26 François M, Osiurak F, Fort A, Crave P, Navarro J. Usability and acceptance of truck dashboards designed by drivers: two participatory design approaches compared to a user-centered design. Int J Ind Ergon 2021; 81: 103073
• 27 Setyawan M, Awangga R, Afriyanti R. Dashboard settings design in SVARA using user-centred design method. Telkomnika 2019; 17 (02) 615-619
• 28 Pestana M, Pereira R, Moro S. Improving health care management in hospitals through a productivity dashboard. J Med Syst 2020; 44 (04) 87
• 29 Yoo DW, Birnbaum ML, Van Meter AR, Ali AF, Arenare E, Abowd GD, De Choudhury M. Designing a clinician-facing tool for using insights from patients' social media activity: iterative co-design approach. JMIR Ment Health 2020; 7 (08) e16969
• 30 Aggelidis VP, Chatzoglou PD. Hospital information systems: measuring end user computing satisfaction (EUCS). J Biomed Inform 2012; 45 (03) 566-579
• 31 Jawa RS, Tharakan MA, Tsai C. et al; Stony Brook Medicine Enterprise Analytics Team. A reference guide to rapidly implementing an institutional dashboard for resource allocation and oversight during COVID-19 pandemic surge. JAMIA Open 2020; 3 (04) 518-522
• 32 Campbell T, Mistry Z, Gorelick-Feldman G, Hodanics C, Babin S, Lewis S. Development of the respiratory disease dashboard for the identification of new and emerging respiratory pathogens. Johns Hopkins APL Tech Dig 2014; 32 (04) 726
• 33 Cheng CK, Ip DK, Cowling BJ, Ho LM, Leung GM, Lau EH. Digital dashboard design using multiple data streams for disease surveillance with influenza surveillance as an example. J Med Internet Res 2011; 13 (04) e85
• 34 Kostkova P, Garbin S, Moser J, Pan W. eds. Integration and visualization public health dashboard: the medi+ board pilot project. Proceedings of the 23rd International Conference on World Wide Web; 2014
• 35 Anantham D, Chai-Lim C, Zhou JX, Phua GC. Operationalization of critical care triage during a pandemic surge using protocolized communication and integrated supportive care. J Intensive Care 2020; 8 (01) 59
• 36 Mok WQ, Wang W, Liaw SY. Vital signs monitoring to detect patient deterioration: an integrative literature review. Int J Nurs Pract 2015; 21 (Suppl. 02) 91-98
• 37 Christen OM, Mösching Y, Müller P, Denecke K, Nüssli S. Dashboard visualization of information for emergency medical services. Stud Health Technol Inform 2020; 275: 27-31
• 38 Meng Y, Zhang Y, Wang S. et al. Lessons learned in the development of a web-based surveillance reporting system and dashboard to monitor acute febrile illnesses in Guangdong and Yunnan provinces, China, 2017-2019. Health Secur 2020; 18 (S1): S14-S22
• 39 Madhavan S, Bastarache L, Brown JS. et al. Use of electronic health records to support a public health response to the COVID-19 pandemic in the United States: a perspective from 15 academic medical centers. J Am Med Inform Assoc 2021; 28 (02) 393-401
• 40 Gibson B, Butler J, Schnock K, Bates D, Classen D. Design of a safety dashboard for patients. Patient Educ Couns 2020; 103 (04) 741-747
• 41 Mlaver E, Schnipper JL, Boxer RB. et al. User-centered collaborative design and development of an inpatient safety dashboard. Jt Comm J Qual Patient Saf 2017; 43 (12) 676-685
• 42 Dixit RA, Hurst S, Adams KT. et al. Rapid development of visualization dashboards to enhance situation awareness of COVID-19 telehealth initiatives at a multihospital healthcare system. J Am Med Inform Assoc 2020; 27 (09) 1456-1461
• 43 Ahn J, Campos F, Hays M, DiGiacomo D. Designing in context: reaching beyond usability in learning analytics dashboard design. J Learn Anal 2019; 6 (02) 70-85
• 44 Reeves JJ, Longhurst CA, San Miguel SJ. et al. Bringing student health and well-being onto a health system EHR: the benefits of integration in the COVID-19 era. J Am Coll Health 2020; •••: 1-7
• 45 Holzer KJ, Lou SS, Goss CW. et al. Impact of changes in EHR use during COVID-19 on physician trainee mental health. Appl Clin Inform 2021; 12 (03) 507-517
• 46 Pestana M, Pereira R, Moro S. Improving health care management in hospitals through a productivity dashboard. J Med Syst 2020; 44 (04) 87
• 47 Wilbanks B, Langford P. A review of dashboards for data analytics in nursing. Comput Inform. Nurs 2014; 32 (11) 545-549
• 48 Vitacca M, Vitacca M. Proposal of a multidimensional strategic-management dashboard for use in a rehabilitation respiratory unit. Medicine (Baltimore) 2019; 98 (20) e15728
• 49 Loginova J, Wohland P. How to create an interactive dashboard using R: the example of the Queensland COVID-19 tracker. Aust Population Stud 2020; 4 (02) 39-47
• 50 Hsu SM, Cheng TK, Chang PJ, Chen TY, Lu MH, Yeh HT. Tracking hospital visitors/chaperones during the COVID-19 pandemic. Appl Clin Inform 2021; 12 (02) 266-273
• 51 Kalankesh LR, Nasiry Z, Fein RA, Damanabi S. Factors influencing user satisfaction with information systems: a systematic review. Galen Med J 2020; 9: e1686
• 52 Aristizábal-Torres D, Peñuela-Meneses C, Barrera-Rodríguez A. An interactive web-based dashboard to track COVID-19 in Colombia. Case study: five main cities. Rev Salud Publica (Bogota) 2020; 22 (02) e215

Publication History

Received: 20 June 2021

Accepted: 11 September 2021

Article published online:
08 December 2021