Introduction
In the first article in this series (EBSJ, volume 3, issue 4), we described some basic terminologies related to health economic
studies. In this second and final article, we describe
As with clinical studies, health economic studies vary in scope and quality. Their
usefulness for decision and policy making is linked to their methodological rigor,
accuracy, and generalizability. Thus, critical appraisal is important to put the results
in the context of study quality.
Economic evaluations identify and compare appropriate alternatives of care, their
incremental impact on health outcomes, and their incremental costs. As described in
the first part of the series, there are several types of comprehensive medico-economic
evaluations.
-
Cost-minimization studies consider the cost differences between alternatives of equal effectiveness. This assumes
each alternative is equally effective, which is rarely true.
-
Cost-benefit studies consider both costs and benefits in monetary terms. Expressing benefits in monetary
terms is, however, controversial.
-
Cost-effectiveness studies consider differences in costs and differences in effectiveness, but effectiveness
is measured variably between studies. For comparisons involving the same disease,
condition, or health state, this may be the most meaningful type of evaluation. It
is less helpful for comparing studies involving different diseases.
-
Cost-utility studies consider differences in costs and outcomes for quality-adjusted survival, most often
using the quality-adjusted life year (QALY). Cost-utility studies have the advantage
of providing an incremental cost-effectiveness ratio (ICER) expressed as “cost per
quality-adjusted life year” (cost per QALY) that eases comparison across multiple
studies. This approach incorporates assumptions regarding how each alternative impacts
patient's quality of life.
Each employs different methodologies, potentially complicating critical appraisal,
but some common criteria can be assessed across studies. No standard, universally
accepted method of critical appraisal of economic analyses is currently in use. Across
the available methods, there is agreement on several points of methodological quality.[1]
Assessing the Quality of Economic Evaluations
Assessing the Quality of Economic Evaluations
Several organizations and research groups (e.g., Center for Health Economics, University
of York [http://www.york.ac.uk/che/]; Tufts University Cost Effectiveness Registry [http://research.tufts-nemc.org/cear4/]) are dedicated to excellence in the conduct of health economic evaluation and provide
detailed insights into many aspects of health economic analysis. The Tufts site contains
a registry that allows one to search for cost-effectiveness analyses.
A variety of tools are available to assist new or experienced readers of economic
evaluations, such as the Quality of Health Economic Studies (QHES) checklist,[2]
[3] the Consensus Health Economic Criteria list (CHEC),[4] and the British Medical Journal (BMJ) standards for quality of health economic evaluations.[5] All share emphasis on critically appraising some common domains of economic evaluations.
In one formal comparison of QHES, the BMJ, and CHEC instruments,[6] the following domains were included for assessment in all three:
At least two of the instruments included assessment of the following domains:
Let us look at a few of the concepts related to these domains and how they apply to
evaluation of study quality in a bit more detail.
What Is the Study Question or Objective of the Economic Evaluation?
What Is the Study Question or Objective of the Economic Evaluation?
Specification of the study question should set the stage for assessing the appropriateness
of the economic evaluation design and methods. If the stated study question is to
evaluate the cost-effectiveness of artificial disc replacement but only information
on costs are presented and no comparison to viable alternative treatments is done,
the quality of the study would immediately be suspected and the usefulness to policy
makers is limited.
Does the Study Compare Costs and Outcomes of Meaningful and Appropriate Clinical Alternatives?
Does the Study Compare Costs and Outcomes of Meaningful and Appropriate Clinical Alternatives?
This is one of the first questions to consider. Studies that report only costs or
do not formally compare clinical alternatives are not considered full economic evaluations.
High-quality economic evaluation will clearly state all components of both intervention
and comparator interventions. Meaningful incremental analysis in a full economic study
is only possible if costs and outcomes for competing alternatives for care are reported.
It is crucial that both intervention and comparator be realistic, appropriate reflections
of actual use or care situations. A justification for why specific alternatives are
evaluated should be stated. Studies that do not take such an approach or do not clearly
state the alternatives being compared are not considered of high quality.
Does the Study Justify Its Perspective and Time Horizon Clearly?
Does the Study Justify Its Perspective and Time Horizon Clearly?
In a high-quality economic evaluation, the perspective is clearly stated and the rationale
for its selection justified. As you may remember from the first article in this series,
economic evaluations can take one of several perspectives, which have direct bearing
on the types of costs and outcomes considered, such as a patient, payer, provider,
health system, or society. The types and sources of cost data, measurement of benefits,
and various modeling assumptions are influenced by the choice of perspective. Whatever
the choice of perspective, the authors' choice of parameters should reflect it. If
a societal perspective is chosen, for example, our critical appraisal should address
the extent to which all long-term costs (both direct and indirect costs such as lost
productivity, recovery time, and direct costs to patients) and outcomes to health
systems and patients are included in the model. The perspective must be consistent
with the research question or objective. The data used must be consistent with the
perspective.
Economic evaluations generally employ complex modeling techniques to estimate costs
and outcomes over time. A high-quality study will use a time horizon of at least 10
years to include all relevant sequelae and relapse, reoperation, or recurrence. Shorter
time horizons can also provide valuable information depending on the study objective
and available data. For longer time horizons, preferences for future benefits over
immediate ones should be described in the study assumptions and modeled appropriately,
usually using discounting of costs and benefits beyond 1 year, typically between 3
and 5%.
How Appropriate Are the Data Sources?
How Appropriate Are the Data Sources?
Economic evaluations regularly include data from disparate sources such as clinical
trials, previously published scientific data, surveys collected from patients, or
health care cost and utilization databases. High-quality studies provide clear descriptions
for all data sources, they are complete, justify their selection, and employ data
that are least subject to bias.
-
Patient-level clinical and cost data collected alongside a randomized controlled trial
(RCT) or high-quality meta-analyses for RCTs are examples of data with low potential
for bias. If these data are not available, then data from high-quality observational
studies (e.g., methodologically rigorous prospective cohort studies) can provide useful
information for modeling.
-
Existing databases can provide valuable cost estimates as well, though should be critically
appraised for their generalizability and their fit with the study's perspective. For
example, data from a health plan cost-and-use database used in a study taking a societal
perspective might have incomplete data on indirect costs to patients.
-
Expert opinion—providers' own estimates of clinical or cost estimates—is often biased
and therefore considered the least reliable source of data.
-
Data relevant to all aspects of the economic model need to be described.
How Were Outcomes Measured?
How Were Outcomes Measured?
The choice and measurement of the outcome should reflect the study objective. Instruments
and/or clinical endpoints (e.g., survival) that are reliable and validated in the
patient population of interest are preferred to surrogate or intermediate endpoints.
There is an increasing trend toward the use of patient-important outcomes, ideally
developed with input from the patient population.[7] For spinal conditions, patient-relevant conditions might include pain relief, function,
and productivity.[8]
In a cost-utility analysis, quality-adjusted survival is calculated using a measure
of utility combined with the health outcome in question. A high-quality economic evaluation
will use validated utility measures and state clearly how they were calculated, ideally
from a sample of people affected by condition or from a general community population
(clinician assessment of utility would not be the ideal method of assessment). Some
universal measures used to estimate utility (reported as a number between 0 and 1,
where 1 represents perfect health) include the EQ-5D, the SF-6D, or through patient
weighting of hypothetical scenarios such as discrete choice analysis.[9]
[10] Condition-specific measures are also available, such as the Oswestry Disability
Index.[11]
Were Sensitivity Analyses Presented?
Were Sensitivity Analyses Presented?
Economic evaluations employ a variety of inputs and assumptions, all of which may
induce uncertainty and variability. Thus, a high-quality evaluation will employ sensitivity
analyses to assess uncertainty and evaluate factors that most influence the results.
Some of these techniques can be quite complex. One-way sensitivity analyses vary individual
variables (such as reoperation rate, pain reduction, or procedure cost) across a range
to see how the results are changed. Probabilistic sensitivity analyses employ bootstrapping
or other statistical techniques to arrive at a sense of which variables are “driving”
the model results. This can be useful to decision-makers who need to understand the
most important components of the cost and how they impact the cost-effectiveness.[12]
What about Bias?
Just like in clinical studies, a high-quality economic study will discuss the potential,
magnitude, and direction of potential bias. For example, leaving out indirect costs
to patients in a study taking a societal perspective where indirect costs are higher
in the intervention compared with the alternative might alter the cost-effectiveness
ratio. Failure to account for the range of adverse events can induce bias. Various
assumptions about modeling may introduce bias. For example, if assumptions regarding
the modeling of risk of certain adverse events are from poor quality studies or data
that do not reflect current practice, the results may inaccurately favor one of the
alternatives.
How Transparent Were the Authors?
How Transparent Were the Authors?
Increasingly, health economics practitioners are encouraged to provide the details
of data sources as well as assumptions made regarding modeling of the data to encourage
replication by other teams and so that major assumptions that influence the applicability
of the model and study findings can be examined. The potential for conflict of interest
has become increasingly important among policy makers. High-quality economic study
will clearly state the funding sources provided to conduct the work and the role of
the funder in analysis and reporting of the study.
Are the Conclusions Appropriate?
Are the Conclusions Appropriate?
As in assessing any clinical study, the assessment of its conclusions can be somewhat
subjective but should be based on clinical knowledge and can be somewhat of an art.
In addition to the points presented earlier, a more “gestalt” level assessment can
consider how well
-
the study objective matched the study design and methods employed; as well as
-
the strength and certainty of the justification of the conclusions given the types
of data used, modeling assumptions, perspective, time horizon, and potential sources
of bias.
In other words, if the conclusions of a study seem too good to be true, then they
just might be.
Summary
There are many opportunities for bias in economic evaluations. Thus, critical appraisal
of economic studies is very important but complex. No standard methodology for appraisal
has been universally adopted. However, consideration of the common elements that constitute
a high-quality economic study provides the clinician with the tools to understand
when one might be dealing with a high-quality study, a deeply flawed one, or one that
made the best use of the data available.
Studies of economic impact have become increasingly important in health policy decisions
given the finite resources available for health care. While they should not be the
sole basis of decision making, such studies provide potentially valuable information
if well done. It is therefore important that high-quality economic studies be conducted
and the results of economic studies be placed in the context of their quality.
Learn more about economic evaluation
-
Drummond MF, Sculpher MJ, Torrance GW, O'Brien BJ, Stoddart GL. Methods for the Economic
Evaluation of Health Care Programs. New York, NY: Oxford University Press; 2005
-
What is an economic evaluation? Introduction to Economic Evaluation. Centers for Disease
Control and Prevention. Available at: http://www.cdc.gov/owcd/EET/SeriesIntroduction/1.html
-
Health Economics Information Resources. A self-study course. National Information
Center on Health Services Research and Health Care Technology (NICHSR). Available
at: http://www.nlm.nih.gov/nichsr/edu/healthecon/beginningend.pdf
-
Boos N. The impact of economic evaluation on quality management in spine surgery.
Eur Spine J 2009;18(Suppl 3):338–347
-
Holtz A, Nelson HD, Reid E. Our questions, our decisions: standards for patient-centered
outcomes research. Preliminary Draft Methodology Report; PCORI Methodology Committee;
Patient-Centered Outcomes Research Institute 2012; Available at: http://www.pcori.org/assets/Preliminary-Draft-Methodology-Report.pdf
