The Dangers of Gathering Data: Surgeon-Specific Outcomes Revisited

ABSTRACT

The accuracy of risk adjustment is important in developing surgeon profiles. As surgeon profiles are obtained from observational, nonrandomized data, we hypothesized that selection bias exists in how patients are matched with surgeons and that this bias might influence surgeon profiles. We used the Society of Thoracic Surgeons risk model to calculate observed to expected (O/E) mortality ratios for each of six cardiac surgeons at a single institution. Propensity scores evaluated selection bias that might influence development of risk-adjusted mortality profiles. Six surgeons (four high and two low O/E ratios) performed 2298 coronary artery bypass grafting (CABG) operations over 4 years. Multivariate predictors of operative mortality included preoperative shock, advanced age, and renal dysfunction, but not the surgeon performing CABG. When patients were stratified into quartiles based on the propensity score for operative death, 83% of operative deaths (50 of 60) were in the highest risk quartile. There were significant differences in the number of high-risk patients operated upon by each surgeon. One surgeon had significantly more patients in the highest risk quartile and two surgeons had significantly less patients in the highest risk quartile (p < 0.05 by chi-square). Our results show that high-risk patients are preferentially shunted to certain surgeons, and away from others, for unexplained (and unmeasured) reasons. Subtle unmeasured factors undoubtedly influence how cardiac surgery patients are matched with surgeons. Problems may arise when applying national database benchmarks to local situations because of this unmeasured selection bias.

KEYWORDS

Cardiovascular operations - outcomes - complications - risk adjustment

REFERENCES

• 1 Shahian D M, Normand S L, Torchiana D F et al.. Cardiac surgery report cards: comprehensive review and statistical critique.  Ann Thorac Surg. 2001;  72 (6) 2155-2168
• 2 Grunkemeier G L, Zerr K J, Jin R. Cardiac surgery report cards: making the grade.  Ann Thorac Surg. 2001;  72 (6) 1845-1848
• 3 Schneider E C, Epstein A M. Use of public performance reports: a survey of patients undergoing cardiac surgery.  JAMA. 1998;  279 (20) 1638-1642
• 4 Marshall M N, Shekelle P G, Leatherman S, Brook R H. The public release of performance data: what do we expect to gain? A review of the evidence.  JAMA. 2000;  283 (14) 1866-1874
• 5 Thomas J W, Hofer T P. Accuracy of risk-adjusted mortality rate as a measure of hospital quality of care.  Med Care. 1999;  37 (1) 83-92
• 6 Naftel D C. Do different investigators sometimes produce different multivariable equations from the same data?.  J Thorac Cardiovasc Surg. 1994;  107 (6) 1528-1529
• 7 Edwards F H, Clark R E, Schwartz M. Coronary artery bypass grafting: the Society of Thoracic Surgeons National Database experience.  Ann Thorac Surg. 1994;  57 (1) 12-19
• 8 Hattler B G, Madia C, Johnson C et al.. Risk stratification using the Society of Thoracic Surgeons Program.  Ann Thorac Surg. 1994;  58 (5) 1348-1352
• 9 Edwards F H, Grover F L, Shroyer A L, Schwartz M, Bero J. The Society of Thoracic Surgeons National Cardiac Surgery Database: current risk assessment.  Ann Thorac Surg. 1997;  63 (3) 903-908
• 10 Ghali W A, Ash A S, Hall R E, Moskowitz M A. Statewide quality improvement initiatives and mortality after cardiac surgery.  JAMA. 1997;  277 (5) 379-382
• 11 Ivanov J, Tu J V, Naylor C D. Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery.  Circulation. 1999;  99 (16) 2098-2104
• 12 Breiman L. Bagging predictors.  Mach Learn. 1996;  24 123-140
• 13 Blackstone E H. Breaking down barriers: helpful breakthrough statistical methods you need to understand better.  J Thorac Cardiovasc Surg. 2001;  122 (3) 430-439
• 14 D'Agostino Jr R B. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group.  Stat Med. 1998;  17 (19) 2265-2281
• 15 Rosenbaum P R, Rubin D B. Reducing bias in observational studies using subclassification on the propensity score.  J Am Statist Assoc. 1984;  79 516-524
• 16 Iezzoni L I. Risk Adjustment for Measuring Health Care Outcomes. 3rd ed. Chicago: Health Administration Press; 2003
• 17 Shahian D M, Torchiana D F, Normand S L. Implementation of a cardiac surgery report card: lessons from the Massachusetts experience.  Ann Thorac Surg. 2005;  80 (3) 1146-1150
• 18 Goldstein H, Spiegelhalter D J. League tables and their limitations: statistical issues in comparisons of institutional performance (with discussion).  J R Stat Soc [Series A]. 1996;  159 385-443
• 19 Donabedian A, Bashshur R. An Introduction to Quality Assurance in Health Care. Oxford; New York: Oxford University Press; 2003
• 20 Kohn L T, Corrigan J, Donaldson M S. Institute of Medicine. (U.S.). Committee on Quality of Health Care in America. To Err is Human: Building a Safer Health System. Washington, D.C.: National Academy Press; 2000
• 21 Birkmeyer J D, Dimick J B, Birkmeyer N J. Measuring the quality of surgical care: structure, process, or outcomes?.  J Am Coll Surg. 2004;  198 (4) 626-632
• 22 Shahian D M, Edwards F H, Ferraris V A Society of Thoracic Surgeons Quality Measurement Task Force et al.. Quality measurement in adult cardiac surgery: part 1—conceptual framework and measure selection.  Ann Thorac Surg. 2007;  83 (4, Suppl) S3-S12
• 23 O'Connor G T, Plume S K, Olmstead E M The Northern New England Cardiovascular Disease Study Group et al.. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery.  JAMA. 1996;  275 (11) 841-846
• 24 Grover F L, Johnson R R, Shroyer A L, Marshall G, Hammermeister K E. The veterans affairs continuous improvement in cardiac surgery study.  Ann Thorac Surg. 1994;  58 (6) 1845-1851