Key Performance Indicators Applied in Medicine

Abstract

Introduction

Key performance indicators (KPIs) measure the critical functions and activities that must be performed in an organization to provide management with an assessment of quantity and quality, from which it can aid strategic decisions to achieve objectives.

Objective

To map the literature review on KPIs in the context of medicine.

Methods

In the literature review, we adopted a proposed model and made appropriate adaptations to it. The PubMed database was chosen, with filters applied to define a path. The findings of the articles were briefly discussed in comparison with others to highlight specific findings.

Results

Almost 100 research articles were eligible for investigation in a wide range of third- and second-degree medical specialties. In these articles, the KPIs are numbered from 1 up to 797 qualitative and quantitative measures.

Conclusions

The KPIs are not limited to a financial regime; instead, in the field of medicine, they play an important role in a wide range of medical specialties, from clinics, therapeutics, surgery, pharmacy, research, regulatory affairs, hospitalization, management, and product development. These indicators can be implemented and evaluated by different means, from simple resources such as home visit, telephone survey, cross-comparison with a benchmark, questionnaires, spreadsheets, to the most sophisticated technology, such as IT resources, artificial intelligence, the linear hinges model, the Research and Development/University of California, Los Angeles (RAND/UCLA) appropriateness method, and the Monte Carlo simulation.

 

Scroll to top

Keywords

health care quality indicators - institutional management teams - quality improvements - health services research - knowledge discovery

Introduction

In a systematic review on measurements and improvements during the coronavirus disease 2019 (COVID-19) pandemic, Amer et al.[1] categorized 797 key performance indicators (KPIs) that resulted in 45 subdimensions: finances, efficiency and effectiveness, availability and quality of manpower and supplies, development of health professionals, evaluation of managerial tasks, responsibilities of health professionals, time metrics, safety and elimination of risk factors, communication, community and reputation, and technological resources.

At a Cancer Detection Centre in Belgium, a 15-year extension of the population-based mammography screening program found that most of the KPIs met the European Union (EU) benchmarks, despite the suboptimal attendance rate. These authors stated that KPIs “cannot replace a mortality analysis, but enable programs to compare performance against objectives”.[2] These indicators are specifically designed and measurable metrics of being used within healthcare to measure performance by acting as ‘flags’ or ‘alerts’ to identify good practice, provide comparability, as well as to identify areas for improvement within a service allocation.[3] This concept was used in one Brazilian study meaning “a quantitative measure that can be used to track an organization's progress and allows the monitoring, decision-making, and quality improvement”.[4]

Emergency medicine has proven that a survival benefit was identified when the response time was up to four minutes for patients at intermediate or high risk of mortality.[5] The adoption of a health service performance framework as a platform for a set of specific, measurable and action-oriented KPIs would be a positive step forward.[6] Simple but important qualitative measurements at emergency care demonstrate that “there is a need for less simplistic quality indicators which recognize that there are many stages between a patient's call for help and safe arrival in hospital”.[7].

In time, KPIs were implemented in several medical specialties, including Anesthesiology,[8] Cardiology,[9] [10] [11] Coloproctology,[3] [12] Community Health Sciences,[13] Dentristy,[14] eHealth,[15] Emergency,[16] [17] [18] Endoscopy,[19] Endocrinology,[20] [21] Epidemilogy and Infection,[22] Genetics,[23] Hospital Management,[24] Intensive Care,[25] Medical Education,[26] Mental Health,[27] Nephrology,[28] Nursery,[29] Nursing and Midwifery,[30] [31] Oncology,[32] Orthopedic Surgeon,[33] Optopmetry,[34] Otolaryngology,[35] [36] Palliative Care,[37] Phatology,[38] Pharmacy,[4] [39] Plastic Surgery,[40] Prosthetic and Orthotic,[41] Primary Healthcare,[42] Psychiatry,[43] Radiology,[44] [45] [46] Reprodution,[47] [48] [49] Speech Pathology,[50] Sports Medicine,[51] [52] [53] Urogynecologist,[54] and Therapeutics Innovation.[55]

Our team has identified tangible KPIs that are not limited to the articles cited in the foreground, but which provide a review of the literature in medicine with restrictions on time, database and type of research, as we will describe later.

Hence, our objectives are to collect a set of articles in medicine in line with KPIs; to carry out a descriptive analysis of the literature on the subject, where applicable; and to summarize elements of its main conclusions, where applicable.

 

Scroll to top

Methods

The present article is organized based on a template designed by Xiao and Watson[56] for literature reviews ([Fig. 1])

Fig. 1 Process of literature review. Note: Adapted from Xiao and Watson.[56]

Step 1–Formulate the Problem

Instead of zooming out too many research questions, we follow the footprints to formulate a specific problem and then look at how to map and solve that question. Cronin et al.[57] suggested a theoretical framework to answer three questions: “Has a conceptual or theoretical framework been identified? Is the framework adequately described? Is the framework appropriate?” Which is what we follow here.

 

Scroll to top

Step 2–Develop and Validate the Review Protocol

In short, it is a predefined plan that specifies the methods for carrying out the review. The review protocol increases the reliability of the research, as readers can use the same protocol to repeat the study for cross-checking and verification.

Our initial research found 7,720 articles. At the second try they were reduced to 1,219 articles, but both cases could be labeled as a large sample for the literature review. Our review protocol needed to move forward for the sake of reasoning.

 

Scroll to top

Step 3–Search for Literature

We value access to full texts more than abstracts; that is why we used the virtual private network (VPN) of Universidade de São Paulo (USP) to search PubMed to ensure a secure connection and access the full texts.

Refining our research led us to filter article type as a set of combinations: Action Research, Cohort, Cross-Sectional, Design of Experiments, Explanatory Design, Exploratory, Gap Analysis, Observational, Narrative Review, Randomized Controlled, Retro-Prospective, Retrospective Comparative, Phenomenological Methodology, Prospective, Prospective Cohort, Longitudinal, and Study Protocol.

We used the filters as defined: Human, English, Male and Female, and, lastly, only MEDLINE. The timeframe was set as 17 September 2019 to 17 September 2023, which can be justified by considering recent research as more relevant to the current situation, therefore being able to provide more useful insights.[56] There were 58 positive results for our criteria at this point. Furthermore, MEDLINE provided tools such as the possibility of sending the search results by e-mail, and just a .txt file with abstracts and a .cvs file with the main components of the search.

 

Scroll to top

Step 4–Screen for Inclusion

Once more, Xiao and Watson[56] paved the way through an efficient two-step procedure: first, a general search of articles for inclusion based on a review of the abstracts, followed by a refined quality assessment based on a full-text review. As our literature search did not cover a large proportion of the medical specialties we originally sought, we were obliged to add another 51 articles.

We noticed that some summaries were blurry, so we had to search in the text itself, for example, when the term Quality is recorded[58] [59] [59] [60] or when a Biomaterial is set.[61] Some researches are still in progress;[62] [63] [64] however, they were still considered for our literature review.

 

Scroll to top

Step 5–Access Quality

In time, two texts were excluded: Halling et al.,[65] as it was not focused on health care or KPIs; and Nicholson's study,[66] which is conceptual. Thus, the final results led to 56 articles used as primary data. Complementary literature was also used to support our work.

 

Scroll to top

Step 6–Extract Data

Our cache of 58 articles ([Supplementary Appendix 1]; online only) from PubMed were organized in a spreadsheet-like format where we preserve authors ‘original own words rather than put our own version to maintain authors’ concepts. On the column Main Findings, you expect to have full authors' findings when that sentence ends with period. In case authors' findings cannot fit completely, we adopt parentheses with ellipsis points inside. We invite the readers to access the full text for a broader comprehension. The authors' text was totally preserved. “Copy and paste” was our data extraction pattern to preserve the authors' fundamentals.

 

Scroll to top

Step 7–Analyze and Synthesize Data

Sandelowski et al.[67] highlights the problems faced when combining qualitative (such as differences in ontological positions, epistemological positions, research paradigms, fundamental theories, philosophies, and methodologies) with quantitative literature (such as the heterogeneity of the studies). Both methods are important, as our study did not aim to cross-compare the articles or to determine which of them is more relevant on the topic of KPIs. Therefore, we omit our opinion.

 

Scroll to top

Step 8–Report Findings

Our primary choice treasures the convenience and clearness of spreadsheet-like format with Article, Declared Area of Interest / Related Sciences, Type of Experiment/Study, KPIs Method, KPIs Quantity, and Main Findings / Results. Our search and filtering flow are shown in [Fig. 2].

Fig. 2 Screening flow of the present article, following the Preferred Reporting Items for Systematic Reviews and Meta-Analyzes (PRISMA) statement.

 

Scroll to top

 

Scroll to top

Results

Descriptive Analyses

Our temporal analysis tells us 2021 was the most prolific year with 31 publications (55%), followed by 2019 with 11 publications (20%), then 2018 with 5 publications (9%). The lowest ratios were found in 2023, 2022, and 2020, with three publications each (5% each), as shown in [Fig. 3].

Fig. 3 Literature evolution.

We don't have the space to discuss each of the final 56 articles, but some deserve a few observations in the light of the KPIs.

 

Scroll to top

Selected Articles (2018)

Two articles have a cross-comparison results with Government or Economic Bloc guidelines: Fitzpatrick et al.[68] and Philippon et al.[69] The former covers the national BreastCheck screening program in the Republic of Ireland over a period of more than a decade, compared with the standards of the program and European guidelines. Authors highlight that the “Adoption of a longitudinal perspective on screening programs and examination of their performance indicators over a sustained period is worthwhile” on anticipating a trend: The uptake of BreastCheck examinations remained above the acceptable standard of 70% throughout the period but has been falling since 2010.

The latter article makes comparative quantitative analyses of the Canadian and Australian health systems to affirm the comparatively superior performance of the Australian system on most indicators, concluding that Canada needs to adopt a more serious approach to the federal role and the introduction of a national pharmaceutical program. In one of them,[69] the authors sent a message to authorities: “It is hoped that this article will empower health leaders to take action in these areas.”

 

Scroll to top

Selected Articles (2019)

Jones et al.[70] developed a mortality and discharge modeling of hospitalized stroke patients using a Markov-based phase recovery (stochastic model used to model systems that change pseudo-randomly) for patients with transient ischemic attack (TIA), noting that “the likelihoods of death and discharge to nursing home increase with age, and the likelihood of discharge to usual residence decreases with age.”

In the Netherlands a published by Dohmen and van Raaij[71] defined 10 KPIs for cataract surgery without flags, just plain values for perioperative complications (maximum: 1%), postoperative complications (maximum: 1.2%), conducted front vitrectomy (maximum: 0.2%), corrected visual acuity (≥ 0,5–96%), patient satisfaction of outpatient visit (≥ 90%), average grade by patients scale from 0 to 10 (≥ 8), net promotor score (≥ 55), availability of operating room (100%), lead time first hospital visit to operation (maximum: 30 days), and patient satisfaction with waiting time (≥ 9 0%).

Prospective research[72] on provisional colonoscopy certification (the transition from training to the new independent practitioner) stated that some KPIs exceeded the national standard of 90%. Trainees with declining performance incurred higher rates of moderate to severe discomfort after provisional colonoscopy certification, despite needing higher dosages of analgesics, and were more likely to need assistance from an instructor in failed procedures. A total of 257,800 colonoscopies were performed over a 6-year period.

 

Scroll to top

Selected Articles (2020)

An observational cohort study in Spain[73] had hypothesized that, due to the lower response to surgical aggression and lower rates of postoperative complications, enhanced recovery protocols after surgery may reduce colorectal cancer-related mortality. However, no updates were found for this research.

Across 16 competitive matches a total of 15 males outfield players (5 defenders, 6 midfielders, and 4 strikers), aged 16 to 18 years, from the club's scholarship squad were subjects to perform fitness testing battery under specific circumstances. Wing et al.[74] were intrigued about the importance of strength and power among these athletes. Results showed that there was a significant correlation between countermovement jump, squat jump, and total score of athleticisms in relation to heading success. In this case, the KPIs were passing, shooting, dribbling, tackling, and heading.

 

Scroll to top

Selected Articles (2021)

Ratnovsky et al.[75] aimed to perform an exploratory dig in data analysis to develop an admission prediction model based on a dataset that was constructed from KPIs selected by a panel of hospital administrators, physicians and nurses. The patients per day indicator was significantly higher on Sundays compared with all other weekdays (p < 0.001) as it did not differ significantly between midweek days. The distribution on Fridays was significantly different from the rest of the weekdays and Saturday. The length of stay of patients' indicator did not differ significantly between Sunday and weekdays (p = 0.288).

The British Orthopedic Association and British Geriatric Society published, in 2007, their care guide for patients with fragility fractures (Blue Book) in which the KPIs for hip fracture care were determined. Crozier-Shaw et al.[76] were drawn to analyze Blue Book compliance during the COVID-19 pandemic and analyze hip fracture outcomes, including the COVID-19 infection rate. They write of a 20% reduction in hip fractures during 2020, compared with 2019, but the pandemic has increased mortality associated with hip fractures, although not in a statistically significant way. They comment on a Chinese experience, particularly of a Wuhan study with 10 hip fractures and COVID-19, which showed a mortality rate of 40%.

One case of biomaterial development was listed on our sample. Photosensitizer nanoparticles emission for precise photodynamic oral cancer therapy as an alternative for drugs. Wu et al.[61] declared their 532 nm clinical laser irradiation could generate 3.78-fold more ¹O2 than clinically approved hemoporfin. Based solely on the mouse model of oral cancer, it was possible to demonstrate that AIEPS5-NPs-NB (material development and related technique) can lead to more than 90% tumor ablation under laser irradiation, and the inhibition rate was much higher than that of hemoporfin.

A tertiary hospital with a capacity of 1,300 beds, 1,654 items in the pharmacy, and 6 robotics systems that can serve 1 thousand patients daily have a related article due to the unpredictable COVID-19 experience. The development of a near real-time pharmacy dashboard can be useful in managing this workload, but it brought obstacles regarding the integration of different database platforms, such as the Oracle Database Server (Oracle Corp.) with SQL Database (Microsoft Corp.), in which the Cloverleaf engine (Infor) was chosen. All IT must comply with the Health Insurance Portability and Accountability Act at all stages of data extraction, transfer, manipulation and aggregation. Al-Jazairi et al.[77] point out that “minimizing therapy interruptions and improving the patient experience—all of which are becoming priorities for large-scale, complex organizations.” Here, we present a collection of 31 KPIs considered.

There were two articles highlighting KPIs in sports to measure athlete performance: one is about swimming[78] and the other regarding soccer.[79] The former measured lifesaving-specific skills (block time with fins, 50m swimming with fins, 25m mannequin carry, and 50m mannequin tow with fins), swimming skills (block time freestyle, 50m freestyle, 15m split time, 25m split time, tumble turn, 200m freestyle, and pace at 4 mmol/L), muscular strength and endurance (ventral core, lateral core, dorsal core, bench press, bench pull, and standing long jump), flexibility (shoulder inside rotation, shoulder outside rotation, hip extension, sit-and-reach, plantar flexion, and supination), and anthropometry data (age, height, arm span, and body mass), with a total of 27 KPIs.[78] The latter took a sample of 145 players from 42 countries, with 64.3% of practitioners using data tools and applications weekly, according to the KPIs. In a general run, 83% of participants reported the use of event data with technology. As 35% of participants reported using their own tagging within platforms, such as Hudl or Sportscode, rather than relying on external sources such as Stats Perform or official league sources such as Bundesliga Event Data. The most used KPIs were shots on goal (77%), shots from penalty area (73%), total shots (70%), crosses (70%), shooting efficiency (68%), shots from the goal box (57%), expected goals (46%), pass evaluation metrics (44%), space control (33%) and expected possession value (22%).[79]

 

Scroll to top

Selected Articles (2022)

Colorectal surgery is still associated with a significant burden of postoperative complications and, ultimately, costs for healthcare. Massa et al.[63] launched an audit to evaluate its impact in a large area of northern Italy. All consecutive patients undergoing surgery for primary colorectal cancer between 15 April 2019 and 31 December 2023 were enrolled. This research is still in progress.

A study protocol by Opgenorth et al.[62] is still underway. Its aim is to better manage acute renal replacement therapy (KRT), which is administered to acutely ill patients to support kidney function and life in the Intensive Care Unit (ICU). Implementing standardized acute KRT pathways can ensure its safe and effective management. The Dialyzing Wisely Critical Care Strategic Clinical Network is a registry-integrated, interrupted time-series evaluation of the implementation of a standardized, stakeholder-informed, evidence-based acute KRT pathway. Dialyzing Wisely will implement, monitor, and report on a set of KPIs, along with a care pathway that will transform the quality of acute KRT in Alberta's ICUs as, at present, there is no standardized approach.

 

Scroll to top

Selected Articles (2023)

In Uganda almost 75% of the entire population is rural, while many healthcare workers are concentrated in urban areas. Minimizing such issues drove them to create community health workers (CHW) programs, with a team of 5 to 6 people as village health teams (VHTs). Agarwal et al.[80] argues that these programs face significant disputes, due to poor retention and insufficient financial investments. Furthermore, they mention threats to personal health, such as the COVID-19 and Ebola pandemics. This study is in progress and is KPIs-related.

The Shanghai Stroke Service System (4S) is a regional stroke network making use of an online database that automatically extracts data from structured electronic medical records. Stroke is one of the leading causes of death and disability in China. Patients who are 18-years or older with International Classification of Diseases 10th Revision (ICD-10) stroke codes (I63, I61, and G45) as the primary diagnosis at discharge were included in the study by Xu et al., with 11 KPIs listed.[58] All stroke and transient ischemic attack (TIA) admissions from participating centers, between January 2015 and December 2020, were included. Sample fetch consisted of 92,395 stroke or TIA admissions from 61 centers. The study concluded that improvement of stroke care can be better achieved by a regional management intervention basis, instead of a centralized one.

 

Scroll to top

 

Scroll to top

Discussion

In the context of educational proceedings, the performance indicators can be traced back to 1880, as noted by Hogg (apud Fitz-Gibbon[81]): “In the late nineteenth century both in Europe and the United States the growth of new methods of scientific measurement, for example the pioneering work of Edgeworth at Oxford in the 1880s, aided the development of tests to assess schoolchildren”.

Articles in the medical field that explore performance indicators date back to 1923. Robinson[82] stated that “to realize the main distinctions between different types of work, it is also important to realize the factors common to all types of human performance”. Later, in 1931, Symonds[83] expressed that measurements are needed to “include it only the measurement of intelligence and achievement, but also the non-intellectual qualities and characteristics variously know as character, conduct, traits, personality, emotionality (...).”

Along with these historical records, the present article makes relevant contributions to the field of medicine by revealing that KPIs are a constant in the practice of medicine. For example, this investigation presents studies with results over 35 medicine specialties ranging from clinical,[9] [12] [28] [42] therapeutics or medication,[10] [63] [84] surgery,[8] [24] [31] [36] laboratory and pharmacy,[4] [39] [77] novelty,[23] [30] [47] [48] regulation, guidelines, and national protocols,[3] [58] [68] [85] [86] [87] hospitalization,[11] [72] [88] exploratory,[13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [25] [27] [29] [32] [80] management,[41] [43] information technology,[44] [89] [90] and equipments[45] [61] and sports.[51] [52] [53] [74] [78] [79]

A second contribution involves carrying out descriptive and thematic analyses together. The descriptive analysis provides a broader understanding of how medical researchers and academics have approached KPIs over time. The evolution of this subject through different methods and applications, in strictly controlled scenarios or in the uncertainty caused by the COVID-19 pandemic, may intrigue discerning readers to develop KPIs, both in medical practice and in the field of research. An in-depth investigation into how to overcome regular and routine obstacles is present here, for regulatory affairs or even for managers and engineers in related fields.

Another contribution of the presented article tells us the KPIs can be implemented and be evaluated by different means, such as household visit, phone survey, cross-comparison with a benchmark, questionary, IT, artificial intelligence, observational, linear hinges model, Research and Development (RAND)/University of California, Los Angeles (UCLA) appropriateness method, and the Monte Carlo simulation.

Private medicine has investors behind a decision-making process in a continuous flow. Return on investments, growth, profit margin, revenue, customer loyalty, inflation, taxes, regulation, competitors maneuvers, technology, mergers, acquisitions, and bankruptcies are all part of the intrinsic vocabulary, like non-medical, related fields (such as consumers goods, capital goods, services).

Regarding the public health sector, where almost none of this vocabulary is used, the Brazilian Unified Health System (Sistema Único de Saúde, SUS, in Portuguese) can be used as an example, as the country considers health a constitutional right. In a nutshell, SUS is universal and free for all in the fifth largest country in the world. Is there room to study and apply KPIs in a public health system like the SUS? Apart from monetary issues, does it make sense to increase efficiency and effectiveness? Is it possible to improve the availability and quality of supplies, as well as the development, training, and performance of health professionals, managerial tasks, safety, communication, reputation, and patient satisfaction in SUS?

Only two articles[35] [36] on otorhinolaryngology, were found, which suggests that KPIs are relatively new to this field. With time, this method will improve diagnosis, prognosis, management, and therapy. Furthermore, artificial intelligence and machine learning can also apply KPIs with the aim of saving lives and time.

Lastly our research team does not propose a method, a guideline, or a model to implement KPIs as a general tool, which could be handy. This could be done in a future study, aiming to humanize KPIs, as it is currently only known as a financial tool.

 

Scroll to top

Limitations to the Study

A superficial view of this subject may lead the reader to look at opportunities to apply for KPIs in their professional routine. With this, the authors emphasize the need for more articles to be discussed with more latitude, more topics, with different points of view. The present could be called a systematic review.

 

Scroll to top

Conclusion

This literature review concerns the use of KPIs in the context of medicine, as analyzed in the areas of clinical practice, therapeutics, surgery, pharmacy, research, regulatory affairs, hospitalization, management, and product development. There is a symmetrical conflict between infinite demand and limited resources, which KPIs can play an important role in bridging, regardless of medical science, wealth or other barriers, as there are many resources to balance the challenge, such as information technology, artificial intelligence, the linear hinges model, the RAND/UCLA appropriateness method, and the Monte Carlo simulation.

 

Scroll to top

 

Scroll to top

Conflict of Interests

The authors have no conflict of interests to declare.

Clinical Essay Registry

Authorization granted and expedited on 5 October 2023; reference number: 6.390.430.

• References

• 1 Amer F, Hammoud S, Khatatbeh H, Lohner S, Boncz I, Endrei D. A systematic review: the dimensions to evaluate health care performance and an implication during the pandemic. BMC Health Serv Res 2022; 22 (01) 621 10.1186/s12913-022-07863-00
• 2 Goossens M, De Brabander I, De Grève J, Van Ongeval C, Martens P, Van Limbergen E, Kellen E. Flemish breast cancer screening programme: 15 years of key performance indicators (2002-2016). BMC Cancer 2019; 19 (01) 1012 10.1186/s12885-019-6230-z
• 3 Morar PS, Hollingshead J, Bemelman W, Sevdalis N, Pinkney T, Wilson G. et al. Establishing Key Performance Indicators [KPIs] and Their Importance for the Surgical Management of Inflammatory Bowel Disease-Results From a Pan-European, Delphi Consensus Study. J Crohns Colitis 2017; 11 (11) 1362-1368 10.1093/ecco-jcc/jjx099
• 4 Lima TM, Aguiar PM, Storpirtis S. Development and validation of key performance indicators for medication management services provided for outpatients. Res Social Adm Pharm 2019; 15 (09) 1080-1087 10.1016/j.sapharm.2018.09.010
• 5 Pons PT, Haukoos JS, Bludworth W, Cribley T, Pons KA, Markovchick VJ. Paramedic response time: does it affect patient survival?. Acad Emerg Med 2005; 12 (07) 594-600 10.1197/j.aem.2005.02.013
• 6 O'Meara P. A Generic Performance Framework for Ambulance Services: An Australian Health Services Perspective. Australas J Paramed 2005; 3: 1-13 10.33151/ajp.3.3.319
• 7 Price L. Treating the clock and not the patient: ambulance response times and risk. Qual Saf Health Care 2006; 15 (02) 127-130 10.1136/qshc.2005.015651
• 8 Zbinden AM. Introducing a balanced scorecard management system in a university anesthesiology department. Anesth Analg 2002; 95 (06) 1731-1738 10.1097/00000539-200212000-00049
• 9 Peng Y, Du X, Li X, Ji J, Wu Y, Gao R, Patel A. Clinical Pathways for Acute Coronary Syndromes in China (CPACS) Investigators. Associations Between Education Level and In-hospital Treatment and Outcomes Among Acute Coronary Syndrome in China. Am J Med Sci 2021; 361 (02) 253-260 10.1016/j.amjms.2020.09.015
• 10 Bocchi EA, Moreira HT, Nakamuta JS, Simões MV, Las Casas AdA, da Costa AR. et al; CLIMB-HF Study Group. Implications for Clinical Practice from a Multicenter Survey of Heart Failure Management Centers. Clinics (Sao Paulo) 2021; 76: e1991 10.6061/clinics/2021/e1991
• 11 Kabbani S, Al Habeeb W, Liew HB, Mohan JC, Ogola E, Sim D, Tsabedze N. Supporting the Management of Patients with Heart Failure within Asia-Pacific, Middle East, and African Countries: A Toolbox for Healthcare Providers. Cardiology 2019; 142 (Suppl. 01) 1-10 10.1159/000496663
• 12 Bhat S, Wells CI, Tan J, Bissett IP, Lill M. Colonoscopy quality indicators in patients with and without prior colonic resection: A single-centre prospective comparative study. Colorectal Dis 2021; 23 (07) 1755-1764 10.1111/codi.15628
• 13 Spackman E, Clement F, Allan GM, Bell CM, Bjerre LM, Blackburn DF. et al. Developing key performance indicators for prescription medication systems. PLoS One 2019; 14 (01) e0210794 10.1371/journal.pone.0210794
• 14 Wong C, Blum IR, Louca C, Sparrius M, Wanyonyi K. A retrospective clinical study on the survival of posterior composite restorations in a primary care dental outreach setting over 11years. J Dent 2021; 106: 103586 10.1016/j.jdent.2021.103586
• 15 Vedlūga T, Mikulskienė B. Stakeholder driven indicators for eHealth performance management. Eval Program Plann 2017; 63: 82-92 10.1016/j.evalprogplan.2017.03.001
• 16 Chen Y, Zhong W, Gong X, Hu H, Yan S, Zhang X. et al. Rationale and design of individualized quality improvement based on the Computer Analysing system to improve Stroke management quality Evaluation (CASE): a multicenter historically controlled study. Trials 2020; 21 (01) 677 10.1186/s13063-020-04598-3
• 17 Woodward T, Hocking J, James L, Johnson D. Impact of an emergency department-run clinical decision unit on access block, ambulance ramping and National Emergency Access Target. Emerg Med Australas 2019; 31 (02) 200-204 10.1111/1742-6723.13110
• 18 Penverne Y, Leclere B, Labady J, Berthier F, Jenvrin J, Javaudin F. et al. Key performance indicators' assessment to develop best practices in an Emergency Medical Communication Centre. Eur J Emerg Med 2018; 25 (05) 335-340 10.1097/MEJ.0000000000000468
• 19 Inoue M, Ragunath K. Quality indicators in Barrett's endoscopy: Best is yet to come. Dig Endosc 2024; 36 (03) 265-273 10.1111/den.14654
• 20 Lin JX, Wen D, Sharma A, van der Werf B, Martin RCW, Harman R. Morbidity following thyroid and parathyroid surgery: Results from key performance indicator assessment at a high-volume centre in New Zealand. ANZ J Surg 2021; 91 (09) 1804-1812 10.1111/ans.17099
• 21 Farag S, Feeney C, Lee V, Nagendran S, Jain R, Aziz A. et al. A ‘Real Life’ Service Evaluation Model for Multidisciplinary Thyroid Eye Services. Front Endocrinol (Lausanne) 2021; 12: 669871 10.3389/fendo.2021.669871
• 22 Impouma B, Wolfe CM, Mboussou F, Farham B, Saturday T, Pervilhac C. et al. Monitoring and evaluation of COVID-19 response in the WHO African region: challenges and lessons learned. Epidemiol Infect 2021; 149: e98 10.1017/S0950268821000807
• 23 Bormann CL, Curchoe CL, Thirumalaraju P, Kanakasabapathy MK, Gupta R, Pooniwala R. et al. Deep learning early warning system for embryo culture conditions and embryologist performance in the ART laboratory. J Assist Reprod Genet 2021; 38 (07) 1641-1646 10.1007/s10815-021-02198-x
• 24 Schretlen S, Hoefsmit P, Kats S, van Merode G, Maessen J, Zandbergen R. Reducing surgical cancellations: a successful application of Lean Six Sigma in healthcare. BMJ Open Qual 2021; 10 (03) e001342 10.1136/bmjoq-2021-001342
• 25 Meliones J. Saving money, saving lives. Harv Bus Rev 2000; 78 (06) 57-62 , 64, 66–67
• 26 Amos S, Wiggins JB, Shaw EK, Hannah Jr WN. The Annual Institutional Review-Key Performance Measures and Processes. J Grad Med Educ 2021; 13 (01) 119-122 10.4300/JGME-D-20-00491.1
• 27 García-Alonso CR, Almeda N, Salinas-Pérez JA, Gutiérrez-Colosía MR, Uriarte-Uriarte JJ, Salvador-Carulla L. A decision support system for assessing management interventions in a mental health ecosystem: The case of Bizkaia (Basque Country, Spain). PLoS One 2019; 14 (02) e0212179 10.1371/journal.pone.0212179
• 28 Toussaint ND, McMahon LP, Dowling G, Soding J, Safe M, Knight R. et al. Implementation of renal key performance indicators: promoting improved clinical practice. Nephrology (Carlton) 2015; 20 (03) 184-193 10.1111/nep.12366
• 29 McCance T, Dickson CAW, Daly L, Boomer CA, Brown D, Lynch B. et al. Implementing person-centred key performance indicators to strengthen leadership in community nursing: A feasibility study. J Nurs Manag 2020; 28 (06) 1443-1452 10.1111/jonm.13107
• 30 Radbron E, Wilson V, McCance T, Middleton R. The experience of staff utilizing data to evaluate and improve person-centred practice: An action research study. J Adv Nurs 2022; 78: 3457-3469 10.1111/jan.15386
• 31 Martin E, Beckmann M, Blythe R, Merollini K, Graves N. Adherence to best practice: Preventing surgical site infection following caesarean section in Australia. Aust N Z J Obstet Gynaecol 2021; 61 (05) 728-734 10.1111/ajo.13347
• 32 Gagliardi AR, Lemieux-Charles L, Brown AD, Sullivan T, Goel V. Barriers to patient involvement in health service planning and evaluation: an exploratory study. Patient Educ Couns 2008; 70 (02) 234-241 10.1016/j.pec.2007.09.009
• 33 Tabu I, Goh EL, Appelbe D, Parsons N, Lekamwasam S, Lee J. et al; FERMAT collaborators and the Global Fragility Fracture Network Hip Fracture Audit Special Interest Group. Service availability and readiness for hip fracture care in low- and middle-income countries in South and Southeast Asia. Bone Jt Open 2023; 4 (09) 676-681 10.1302/2633-1462.49.BJO-2023-0075.R1
• 34 Wang H, Kalloniatis M. Clinical outcomes of the Centre for Eye Health: an intra-professional optometry-led collaborative eye care clinic in Australia. Clin Exp Optom 2021; 104 (07) 795-804 10.1080/08164622.2021.1878821
• 35 Taylor B. Using Key Performance Indicators to Do More with Less in Your Practice. Semin Hear 2016; 37 (04) 301-315 10.1055/s-0036-1594000
• 36 Dort JC, Sauro KM, Chandarana S, Schrag C, Matthews J, Nakoneshny S. et al. The impact of a quality management program for patients undergoing head and neck resection with free-flap reconstruction: longitudinal study examining sustainability. J Otolaryngol Head Neck Surg 2020; 49 (01) 42 10.1186/s40463-020-00437-2
• 37 Sathiananthan MK, Crawford GB, Eliott J. Healthcare professionals' perspectives of patient and family preferences of patient place of death: a qualitative study. BMC Palliat Care 2021; 20 (01) 147 10.1186/s12904-021-00842-y
• 38 Tashkandi SA, Alenezi A, Bakhsh I, AlJuryyan A, AlShehry ZH, AlRashdi S. et al. Clinical laboratory services for primary healthcare centers in urban cities: a pilot ACO model of ten primary healthcare centers. BMC Fam Pract 2021; 22 (01) 105 10.1186/s12875-021-01449-1
• 39 Thorakkattil SA, Nemr HS, Al-Ghamdi FH, Jabbour RJ, Al-Qaaneh AM. Structural and operational redesigning of patient-centered ambulatory care pharmacy services and its effectiveness during the COVID-19 pandemic. Res Social Adm Pharm 2021; 17 (01) 1838-1844 10.1016/j.sapharm.2020.06.017
• 40 Patrinely Jr JR, Walker SH, Glassman GE, Davis MJ, Abu-Ghname A, Khan U. et al. The Importance of Financial Metrics in Physician Funding and Performance Evaluation. Plast Reconstr Surg 2021; 147 (05) 1213-1218 10.1097/PRS.0000000000007882
• 41 Brandt JM. An Operational Perspective of The Changing Prosthetics & Orthotics Landscape. Can Prosthet Orthot J 2021; 4 (02) 35996 10.33137/cpoj.v4i2.35996
• 42 Argaw MD, Fekadu BD, Mamo E, Abebe MG, Rogers D, Demelash A. et al. Implementing a Social Accountability Approach for Maternal, Neonatal, and Child Health Service Performances in Ethiopia: A Pre-Post Study Design. Glob Health Sci Pract 2021; 9 (01) 123-135 10.9745/GHSP-D-20-00114
• 43 Russell V, Loo CE, Walsh A, Bharathy A, Vasudevan U, Looi I, Smith SM. Clinician perceptions of common mental disorders before and after implementation of a consultation-liaison psychiatry service: a longitudinal qualitative study in government-operated primary care settings in Penang, Malaysia. BMJ Open 2021; 11 (06) e043923 10.1136/bmjopen-2020-043923
• 44 Sjoeberg J, Omar A, Bujila R, Nowik P, Olafsson I, Poludniowski G. SU-G-IeP3–15: Cloud Based Monitoring of X-Ray Angiography Equipment Performance Based On Automated Daily Sampling of Key Performance Indicators. Med Phys 2016; 43 (6, Part 27): 3670 10.1118/1.4957065
• 45 Sreedher G, Ho ML, Smith M, Udayasankar UK, Risacher S, Rapalino O. et al. Magnetic resonance imaging quality control, quality assurance and quality improvement. Pediatr Radiol 2021; 51 (05) 698-708 10.1007/s00247-021-05043-6
• 46 Obaro AE, Burling DN, Plumb AA. Colon cancer screening with CT colonography: logistics, cost-effectiveness, efficiency and progress. Br J Radiol 2018; 91 (1090) 20180307 10.1259/bjr.20180307
• 47 Wang HT, Hong PP, Li HY, Zhou W, Li T. Use of a new set of key performance indicators for evaluating the performance of an in vitro fertilization laboratory in which blastocyst culture and the freeze-all strategy are the primary treatment in patients with in vitro fertilization. J Int Med Res 2021; 49 (09) 3000605211044364 10.1177/03000605211044364
• 48 Fabozzi G, Albricci L, Cimadomo D, Amendola MG, Sanges F, Maggiulli R. et al. Blastulation rates of sibling oocytes in two IVF culture media: an evidence-based workflow to implement newly commercialized products. Reprod Biomed Online 2021; 42 (02) 311-322 10.1016/j.rbmo.2020.10.017
• 49 Liu J, Schatzkin E, Omoluabi E, Fajemisin M, Onuoha C, Erinfolami T. et al. Introducing the subcutaneous depot medroxyprogesterone acetate injectable contraceptive via social marketing: lessons learned from Nigeria's private sector. Contraception 2018; 98 (05) 438-448 10.1016/j.contraception.2018.07.005
• 50 Carragher M, Steel G, O'Halloran R, Torabi T, Johnson H, Taylor NF, Rose M. Aphasia disrupts usual care: the stroke team's perceptions of delivering healthcare to patients with aphasia. Disabil Rehabil 2021; 43 (21) 3003-3014 10.1080/09638288.2020.1722264
• 51 Caraballo I, Domínguez R, Felipe JL, Sánchez-Oliver AJ. Key performance indicators of Olympic windsurfers during a World Cup: RS:X class®. J Sports Sci 2022; 40 (23) 2645-2653 10.1080/02640414.2023.2180854
• 52 Elsworthy N, R Blair M, Lastella M. On-field movements, heart rate responses and perceived exertion of lead referees in Rugby World Cup matches, 2019. J Sci Med Sport 2021; 24 (04) 386-390 10.1016/j.jsams.2020.10.001
• 53 Ungureanu AN, Brustio PR, Lupo C. Technical and tactical effectiveness is related to time-motion performance in elite rugby. J Sports Med Phys Fitness 2021; 61 (02) 167-174 10.23736/S0022-4707.20.10998-8
• 54 Dietz HP. Ultrasound imaging of maternal birth trauma. Int Urogynecol J 2021; 32 (07) 1953-1962 10.1007/s00192-020-04669-8
• 55 Chong SSF, Kim M, Limoli M, Limoli M, Obscherning E, Wu P. et al. Measuring Progress of Regulatory Convergence and Cooperation Among Asia-Pacific Economic Cooperation (APEC) Member Economies in the Context of the COVID-19 Pandemic. Ther Innov Regul Sci 2021; 55 (04) 786-798 10.1007/s43441-021-00285-w
• 56 Xiao Y, Watson M. Guidance on Conducting a Systematic Literature Review. J Plann Educ Res 2019; 39 (01) 93-112 10.1177/0739456X17723971
• 57 Cronin P, Ryan F, Coughlan M. Undertaking a literature review: a step-by-step approach. Br J Nurs 2008; 17 (01) 38-43 10.12968/bjon.2008.17.1.28059
• 58 Xu J, Xie Y, Fang K, Wang X, Chen S, Liu X. et al. Effect of the Shanghai Stroke Service System (4S) on the quality of stroke care and outcomes: A prospective quality improvement project. Int J Stroke 2023; 18 (05) 599-606 10.1177/17474930221125993
• 59 Zecchin R, Candelaria D, Ferry C, Ladak LA, McIvor D, Wilcox K. et al. Development of Quality Indicators for Cardiac Rehabilitation in Australia: A Modified Delphi Method and Pilot Test. Heart Lung Circ 2019; 28 (11) 1622-1630 10.1016/j.hlc.2018.08.004
• 60 Wu Y, Li S, Patel A, Li X, Du X, Wu T. et al; CPACS-3 Investigators. Effect of a Quality of Care Improvement Initiative in Patients With Acute Coronary Syndrome in Resource-Constrained Hospitals in China: A Randomized Clinical Trial. JAMA Cardiol 2019; 4 (05) 418-427 10.1001/jamacardio.2019.0897
• 61 Wu W, Shi L, Duan Y, Xu S, Shen L, Zhu T. et al. Nanobody modified high-performance AIE photosensitizer nanoparticles for precise photodynamic oral cancer therapy of patient-derived tumor xenograft. Biomaterials 2021; 274: 120870 10.1016/j.biomaterials.2021.120870
• 62 Opgenorth D, Bagshaw SM, Lau V, Graham MM, Fraser N, Klarenbach S. et al. A study protocol for improving the delivery of acute kidney replacement therapy (KRT) to critically ill patients in Alberta - DIALYZING WISELY. BMC Nephrol 2022; 23 (01) 369 10.1186/s12882-022-02990-6
• 63 Massa I, Ghignone F, Ugolini G, Ercolani G, Montroni I, Capelli P. et al; ESCA Collaborative Group. Emilia-Romagna Surgical Colorectal Cancer Audit (ESCA): a value-based healthcare retro-prospective study to measure and improve the quality of surgical care in colorectal cancer. Int J Colorectal Dis 2022; 37 (07) 1727-1738 10.1007/s00384-022-04203-w
• 64 Opgenorth D, Reil E, Lau V, Fraser N, Zuege D, Wang X. et al. Improving the quality of the performance and delivery of continuous renal replacement therapy (CRRT) to critically ill patients across a healthcare system: QUALITY CRRT: a study protocol. BMJ Open 2022; 12 (02) e054583 10.1136/bmjopen-2021-054583
• 65 Halling B, Bergman M, Wijk K. Sense of Coherence and Lean-based leadership and alterations in sick leave and productivity at a steel wire manufacturing unit. Work 2021; 68 (04) 1211-1220 10.3233/WOR-213450
• 66 Nicholson A. Towards a more positive view of healthcare in Ireland. Ir J Med Sci 2019; 188 (01) 35-41 10.1007/s11845-018-1830-1
• 67 Sandelowski M, Voils CI, Barroso J. Defining and Designing Mixed Research Synthesis Studies. Res Schools 2006; 13 (01) 29
• 68 Fitzpatrick PE, Greehy G, Mooney MT, Flanagan F, Larke A, Connors A, O'Doherty A. Evolution of the National Breast Screening Programme in Ireland: Two-year interval analysis (2004-2013) of BreastCheck. J Med Screen 2018; 25 (04) 191-196 10.1177/0969141317738034
• 69 Philippon DJ, Marchildon GP, Ludlow K, Boyling C, Braithwaite J. The comparative performance of the Canadian and Australian health systems. Healthc Manage Forum 2018; 31 (06) 239-244 10.1177/0840470418788378
• 70 Jones B, McClean S, Stanford D. Modelling mortality and discharge of hospitalized stroke patients using a phase-type recovery model. Health Care Manag Sci 2019; 22 (04) 570-588 10.1007/s10729-018-9446-6
• 71 Dohmen PJG, van Raaij EM. A new approach to preferred provider selection in health care. Health Policy 2019; 123 (03) 300-305 10.1016/j.healthpol.2018.09.007
• 72 Siau K, Hodson J, Valori RM, Ward ST, Dunckley P. Performance indicators in colonoscopy after certification for independent practice: outcomes and predictors of competence. Gastrointest Endosc 2019; 89 (03) 482-492.e2 10.1016/j.gie.2018.07.032
• 73 Ramirez-Rodriguez JM, Martinez-Ubieto J, Muñoz-Rodes JL, Rodriguez-Fraile JR, Garcia-Erce JA, Blanco-Gonzalez J. et al. Surgical treatment for colorectal cancer: analysis of the influence of an enhanced recovery programme on long-term oncological outcomes-a study protocol for a prospective, multicentre, observational cohort study. BMJ Open 2020; 10 (10) e040316 10.1136/bmjopen-2020-040316
• 74 Wing CE, Turner AN, Bishop CJ. Importance of Strength and Power on Key Performance Indicators in Elite Youth Soccer. J Strength Cond Res 2020; 34 (07) 2006-2014 10.1519/JSC.0000000000002446
• 75 Ratnovsky A, Rozenes S, Bloch E, Halpern P. Statistical learning methodologies and admission prediction in an emergency department. Australas Emerg Care 2021; 24 (04) 241-247 10.1016/j.auec.2020.11.004
• 76 Crozier-Shaw G, Hughes AJ, Conlon B, Sheehan E, Merghani K. Hip fracture care during Covid-19: a regional trauma centre's experience. Ir J Med Sci 2021; 190 (04) 1275-1280 10.1007/s11845-020-02476-0
• 77 Al-Jazairi AS, Horanieh BK, Alswailem OA. The usefulness of an ambulatory care pharmacy performance dashboard during the COVID-19 pandemic in a complex tertiary care system. Am J Health Syst Pharm 2021; 78 (09) 813-817 10.1093/ajhp/zxab049
• 78 Reichmuth D, Olstad BH, Born DP. Key Performance Indicators Related to Strength, Endurance, Flexibility, Anthropometrics, and Swimming Performance for Competitive Aquatic Lifesaving. Int J Environ Res Public Health 2021; 18 (07) 3454 10.3390/ijerph18073454
• 79 Herold M, Kempe M, Bauer P, Meyer T. Attacking Key Performance Indicators in Soccer: Current Practice and Perceptions from the Elite to Youth Academy Level. J Sports Sci Med 2021; 20 (01) 158-169 10.52082/jssm.2021.158
• 80 Agarwal S, Tweheyo R, Pandya S, Obuya E, Kiyomoto A, Mitra P. et al. Impact of a recognition package as an incentive to strengthen the motivation, performance, and retention of village health teams in Uganda: a study protocol for a cluster randomized controlled trial. Trials 2023; 24 (01) 428 10.1186/s13063-023-07426-6
• 81 Fitz-Gibbon CT. , Ed. Performance indicators. Clevedon, Avon, England; Philadelphia: Multilingual Matters; 1990. . 111 p. (BERA dialogues).
• 82 Robinson ES. Factors Affecting Human Efficiency. Ann Am Acad Pol Soc Sci 1923; 110: 94-104 10.1177/000271622311000110
• 83 Symonds PM. Needed Research in Diagnosing Personality and Conduct. J Educ Res 1931; 24 (03) 175-187 10.1080/00220671.1931.10880196
• 84 Peng Y, Du X, Li X, Ji J, Wu Y, Gao R. et al; Clinical Pathways for Acute Coronary Syndromes in China (CPACS) Investigators. Association of renal insufficiency with treatments and outcomes in patients with acute coronary syndrome in China. Int J Cardiol 2021; 323: 7-12 10.1016/j.ijcard.2020.08.022
• 85 Nag N, Tran L, Fotis K, Smith JA, Shardey GC, Baker RA, Reid CM. Structured Feedback: Acceptability and Feasibility of a Strategy to Enhance the Role of a Clinical Quality Registry to Drive Change in Cardiac Surgical Practice. Heart Lung Circ 2019; 28 (08) 1253-1260 10.1016/j.hlc.2018.07.005
• 86 Network for Peri-operative Critical care (N4PCc)*. Addressing priorities for surgical research in Africa: implementation of a multicentre cloud-based peri-operative registry in Ethiopia. Anaesthesia 2021; 76 (07) 933-939 10.1111/anae.15394
• 87 Practice Committees of the American Society for Reproductive Medicine and Society of Reproductive Biologists and Technologists. Electronic address: [e-mail]. A review of best practices of rapid-cooling vitrification for oocytes and embryos: a committee opinion. Fertil Steril 2021; 115 (02) 305-310 10.1016/j.fertnstert.2020.11.017
• 88 Al Mutair A, Al Mutairi A, Schwebius D. The retention effect of staff education programme: Sustaining a decrease in hospital-acquired pressure ulcers via culture of care integration. Int Wound J 2021; 18 (06) 843-849 10.1111/iwj.13586
• 89 Elamin MB, Flynn DN, Bassler D, Briel M, Alonso-Coello P, Karanicolas PJ. et al. Choice of data extraction tools for systematic reviews depends on resources and review complexity. J Clin Epidemiol 2009; 62 (05) 506-510 10.1016/j.jclinepi.2008.10.016
• 90 Crozier-Shaw G, Hughes AJ, Cashman J, Synnott K. Instant messaging apps and data protection: combining to improve hip fracture care?. Ir J Med Sci 2022; 191 (02) 765-769 10.1007/s11845-021-02612-4

Publication History

Received: 02 October 2024

Accepted: 11 March 2025

Article published online:
26 September 2025