Reliable statistical differentiation of random from nonrandom variation typically requires large amounts of data from broadly representative and diverse populations. Such data were generally unavailable until the mid-20th century. Enactment of Medicare legislation in 1965, though primarily a federal healthcare insurance program, produced a secondary but critical benefit—an expansive set of granular, nationally representative, patient-level health data that could be used for large-scale national quality studies, at least for the older population. Later, beginning in the late 1980s, clinical registry data were introduced by states (e.g., New York), the Veterans Affairs Administration, regional quality initiatives (e.g., the Northern New England Cardiovascular Disease Study Group), and professional societies (e.g., the STS, the American College of Surgeons). These provided standardized, structured, granular, clinically relevant data sources for risk-adjusted outcomes analyses.

The advent of modern computing was also critical for managing large datasets and complex statistical analyses. The availability of mainframe and desktop computers greatly enhanced data storage capacity, computational speed, and portability, thereby allowing large databases to be analyzed efficiently.

Another facilitator of contemporary quality measurement was the evolution and application of modern statistical techniques. Cornfield, in response to Framingham Heart Study investigators, introduced logistic regression in 1961, and variations of this model has been widely used for decades to estimate risk-adjusted binary healthcare outcomes.

In the late 20th century, hierarchical mixed-effect models were first applied to assess education effectiveness in the United Kingdom and for health care ( small area analyses) . Similar approaches were subsequently applied to healthcare provider profiling, where these models explicitly accounted for clustered data (i.e., the patients of a given provider tend to be more similar to one another than to patients of another provider, thereby reducing the number of independent observations); variable and often small sample sizes; and the need to explicitly differentiate random from systematic variation.

For patient observation periods, early studies used in-hospital mortality, which is captured with high accuracy but results in varying postprocedure or postdischarge time intervals, complicating the statistical analyses and contributing to measurement bias. Duration of the index hospitalization can vary substantially depending on the local and regional availability of home care, rehabilitation, extended care, and skilled nursing facilities, and this variation may be completely independent of the quality of care delivered by the index hospital or physician. In areas where these postacute resources are less readily available, patients tend to remain hospitalized for longer initial periods postprocedure, and thus any complications are more likely to be recorded during the index hospitalization. Pouw and colleagues found that index hospitalization length of stay and in-hospital mortality were positively correlated, whereas in-hospital and post-discharge mortality were inversely associated, which they refer to as “discharge bias.” In areas where postacute resources are available, patients are discharged earlier in their recovery, and subsequent death or complications after index hospitalization discharge will not be recorded in in-hospital statistics, a form of ascertainment or surveillance bias. This approach may also lead to intentional gaming or unintentional biasing of the reporting system, in which seriously ill, preterminal patients are discharged and transferred to hospice or other postacute facilities prior to their death. Their subsequent early postdischarge death is not recorded as in-hospital mortality at the index institution.

Given these deficiencies, 30-day postevent (e.g., acute myocardial infarction), 30-day postprocedure (e.g., CABG), or 30-day postdischarge (readmission measures) outcomes are now generally regarded as the gold standard, as they capture all events within this standardized time interval regardless of whether they occur in-hospital, at home, or in extended care. However, like in-hospital metrics, they are also potentially susceptible to gaming, such as keeping a terminally ill patient alive until day 31 before removing life support, thus avoiding a 30-day mortality. Although a theoretical concern supported by many anecdotes, analyses of discharge patterns in states with 30-day versus 30-day plus in-hospital outcomes reporting do not support significant occurrence of this gaming phenomenon. Despite their advantages, 30-day standardized outcomes are more difficult to obtain. For example, even the simplest outcome, 30-day mortality, requires either evidence of a clinic visit or lab test on or after 30 days or direct communication with the patient, their family, or their local physicians. Ascertaining other adverse outcomes, such as sternal wound infection, requires even greater scrutiny.

To mitigate the limitations of in-hospital and 30-day mortality, STS comprehensively includes all mortalities using an endpoint referred to as operative mortality , which includes all index admission deaths regardless of when they take place, and all 30-day deaths regardless of where they occur. Some argue that 30 days is too short in current cardiac surgical practice for assessing the acute outcomes of the index condition or procedure. Ideally, with linkages to other data sources such as the NDI or Medicare claims, longer-term survival, reinterventions, and readmissions will be possible to ascertain consistently and accurately.

The time window over which a provider’s performance is estimated is generally 1 year at a minimum, and more often 3 years to increase sample size, reliability, and power. The original STS CABG Composite measure, ^, which was based on 1 year of data and 98% credible intervals, was updated in 2021 to include 3 years of data and 95% credible intervals (see online Appendix 8A ). These changes resulted in substantial increases in the percentages of both better and worse-than-expected outliers. Three-year samples are typically updated each year with an additional year of data.

In-hospital or 30-day mortality, or combinations thereof (e.g., STS “operative” mortality), remain the most common surgical outcome metrics, but for many procedures, mortality is quite low. As recommended three decades ago by Kouchoukos and colleagues in their STS response to the HCFA mortality reports, outcome evaluations should also include serious postoperative complications, some of which (e.g., disabling stroke or permanent, dialysis-dependent renal failure) are equally or more feared and consequential for patients.

Postoperative hospital readmissions may reflect quality of care during the index admissions. Among surgical patients, including cardiac surgical patients, readmissions also often reflect the initial but delayed, postdischarge appearance of complications, especially as short index hospitalization lengths of stay have been incentivized.

In a study of New York patients who had undergone CABG, Hannan and colleagues found that 84.5% of readmissions were related to CABG complications, of which infections were the most common. The Cardiothoracic Surgical Trials Network (CTSN), sponsored by the National Heart, Lung, and Blood Institute and the Canadian Institutes of Health Research, followed 5158 patients from 10 hospitals prospectively for 60 days after cardiac surgery, and 237 (4.6%) experienced 301 major infections, with 45% occurring after hospital discharge. ^, Mortality was 5% among those infected versus 0.7% among noninfected patients. Major infections accounted for 16% of all readmissions during that 60-day period. These data all suggest that in addition to index hospitalization mortality and complications, postdischarge 30-day readmissions provide a valuable additional window into surgical quality and are not just a reflection of inadequate postdischarge coordination.

Readmission as a metric assumed even greater importance with the advent in 2012 of the CMS Hospital Readmissions Reduction Program (HRRP), which penalizes hospitals financially if their readmissions exceed expected values based on predicted models. Although reduction of readmission is desirable, some have expressed concern that an excessive focus on readmission penalties may lead some hospitals to delay or defer readmissions for patients who would benefit from rehospitalization, and that this might lead to higher overall mortality. Thus, these measures should always be considered in conjunction with balancing measures , such as 30-day mortality. Further, readmission measures may disproportionately affect providers caring for vulnerable populations who have less access to community resources that might prevent unnecessary readmissions. To mitigate this concern, HRRP performance and penalties are now based on peer group stratification (proportion of dual eligible patients).

Process measures typically quantify the proportion of a provider’s patients in which specific, evidence-based, guideline-recommended care processes were used. In cardiac surgery, the STS CABG composite measure includes two evidence-based process measures—use of at least one internal thoracic artery (ITA) bypass conduit and the use of all four NQF-endorsed perioperative medications. Unlike outcome measures, for which an improvement pathway may not be immediately apparent, process measure deficiencies are readily actionable and suggest their own remediation (e.g., use more ITA grafts).

Notwithstanding these advantages, process measures also have their issues, one of the most common being an inadequate evidence base. ^, Further, although these measures are usually not risk-adjusted, they often have substantial exceptions or exclusions that make a substantial proportion of patients ineligible. Thus, unlike outcome measures that have a result for every patient, the subpopulation of patients for whom a particular process measure is applicable may not be representative of the provider’s overall cohort of patients. Other process measure issues include weak process–outcomes associations; “check the box” mentality; ceiling effects or topped-out measures; and poor postdischarge patient compliance. Many object to process measures as being “cookbook medicine” that marginalizes individual physician judgment. Others note that process measures focus attention solely on measured processes of care to the detriment of equally important but unmeasured processes of care. Finally, in some instances, there may be direct or indirect harm. For example, by incentivizing providers to apply process measures to all patients without obvious contraindications, some patients will receive care that may be inappropriate or inconsistent with their wishes.

Given all these considerations, Masoudi and colleagues recommend using clinical guidelines and associated process performance measures with the strongest and most generalizable evidence base; affirmation of trial results in broader populations when data are available; personalized physician judgment when applying guidelines to specific patients; and appropriate use and documentation of exclusions to avoid any incentive for overuse.

Structure measures reflect provider characteristics that are thought to be associated with higher quality care, such as 24/7 intensivist coverage in the ICU, EHRs, clinical registry participation, and appropriate nurse staffing ratios. In surgery, since the pioneering work of Luft and colleagues in 1979, the most widely used structural measure of quality has been provider volume. It has been demonstrated repeatedly that for complex operations, there is an association between volume and outcomes, although the strength of that association varies widely. Volume–outcomes associations also vary depending on methodologic considerations. Studies using more appropriate multilevel, hierarchical approaches tend to show weaker associations.

In adult cardiac surgery, modern volume–outcomes studies have generally shown that these associations do exist and are strongest for more complex, less frequently performed, and newly introduced procedures. Such outcomes associations have led many to call for regionalization of care to higher volume centers, especially for more complex procedures. However, some smaller programs have consistently excellent results over many years, so a combination of volume thresholds and direct outcomes measurement may be fairest.

The Leapfrog Group has now included minimum annual hospital/surgeon volume standards for 11 procedures, many of them in cardiothoracic surgery, including esophageal resection for cancer (20/7); lung resection for cancer (40/15); mitral repair/replacement (40/20); open aortic procedures (10/7); and Norwood procedures (8/5). Implementation of such volume thresholds is not straightforward and there are many nuances. For example, should the same standards apply to highly experienced surgeons who have already performed hundreds or thousands of a particular procedure and are now trying to direct routine cases to younger surgeons? Further, despite the overall volume–outcomes association, there are some low-volume programs that consistently achieve excellent outcomes, and conversely, high volume is not always a guarantee of superior results. Finally, some data suggest that regionalization may not be preferred by all patients. Even when a regional referral center may have better outcomes, patients often prefer local care.

Underuse, misuse, and overuse are three commonly used categories of poor healthcare quality. Underuse is failure to use appropriate, evidence-based testing or treatment, such as routine screening colonoscopy or mammography. This may be especially problematic among minorities and other vulnerable or underinsured populations. Misuse refers mainly to medical errors, avoidable complications, safety events, and other adverse outcomes, brought to public attention most notably in the landmark Harvard Medical Practice Study, ^, the subsequent Institute of Medicine monograph To Err Is Human, (which signaled the beginning of the patient safety movement in US health care), and the 2023 study by Bates and colleagues investigating the current safety of inpatient care. Overuse , a third category of poor-quality health care, refers to care that is inconsistent with evidence-based guidelines, not indicated, or low-value, such as magnetic resonance imaging for uncomplicated low back pain or antibiotics for viral infections.

Regarding overuse, the most systematic approaches to identifying and mitigating this practice are Appropriateness Use Criteria (AUC), which have also been adapted as quality indicators. Cardiac surgery and interventional cardiology have been leaders in the development and application of AUC, contributing three versions for coronary revascularization between 2009 and 2017. These were developed based on expert consensus grading of numerous real-life clinical scenarios, with each scenario scored appropriate (score 7-9), uncertain (may be appropriate, score 4-6), and inappropriate (or rarely appropriate, score 1-3). The value of AUCs as performance metrics has been widely discussed and debated, with limitations including poor documentation and coding; limited number of experts developing AUC decisions; gaming of angina severity; disregard for patient preferences; stress tests not performed or uninterpretable; uncertain vs. inappropriate classification; and physician resentment. A similar AUC for treatment of severe aortic stenosis was published in 2017.

When the multiorganizational AUC criteria for coronary revascularization have been applied to state and national populations who have undergone these procedures, some results have raised concerns. Studies of acute percutaneous coronary intervention (PCI) for myocardial infarction have shown these procedures to almost always be appropriate, with similar findings for virtually all CABG procedures. Conversely, in several studies, roughly half of PCIs for nonacute patient presentations were found to be uncertain or inappropriate. The value of calculating and publicizing such appropriateness data is demonstrated by the substantial longitudinal improvements in nonacute PCI appropriateness found by Hannan and colleagues in New York and Desai and colleagues in a study of 766 centers in the National Cardiovascular Data Registry. Similarly, a study of the appropriateness of surgical AVR for severe aortic stenosis showed occurrence of “rarely” or “may be appropriate” of 3.83% before release of national AUC, declining to 2.06% after their publication.

Most quality measures are defined, scored, and reported by physicians, but some outcomes are best provided directly by patients without physician filtering or interpretation. These include various combinations of general or condition-specific physical symptoms, pain, mental and emotional state, functional status, disability, health-related quality of life, and health behaviors, collectively referred to as patient-reported outcomes. Patient-reported adverse outcomes not only reflect the unfiltered perspective of patients but often occur with greater frequency than commonly reported clinical outcomes such as death or major complications, and may thus better differentiate performance. They also reflect what is most important to the patient, which may sometimes be quite different from the provider’s priorities.

Patient-reported outcome measures may be used longitudinally to compare a patient’s preoperative and postoperative status in the same functional domains, with patients serving as their own control; to compare the course of an individual patient with that of “average” patients to identify potential areas of opportunity for recovery and rehabilitation; and to serve as clinical trial endpoints. In some instances, they may also be useful to compare the functional or symptomatic recovery of groups of patients cared for by different physicians or surgeons (for example, the rate of return of joint function after total knee or hip replacement by different surgeons), although this role as patient-reported outcomes-based performance measures (PRO-PMs) remains controversial. ^, Issues include the rationale for using a PRO-PM in specific contexts; identifying and applying measures that are most meaningful to patients, actionable, and that have acceptable psychometric properties; ensuring measure sensitivity to differences in mode, method, time, place, and frequency of administration; whether and when to risk adjust (e.g., in cross-sectional analyses, when providers care for markedly different populations); and establishing a framework for implementation, scoring and stakeholder interpretation, broad dissemination, and ongoing refinement.

Patient-reported outcomes may be generic, focusing on a broad spectrum of domains, such as the SF-36 or SF-12, or the NIH-funded Patient-Reported Outcomes Measurement Information System (PROMIS), ^, which uses item response theory and computer adaptive testing to measure physical, mental, and social well-being and health-related quality of life domains (e.g., pain, fatigue, depression, and physical function). In cardiovascular disease, the New York Heart Association (NYHA) class has long been used to characterize cardiac symptoms, with newer and more comprehensive measures including the Kansas City Cardiomyopathy Questionnaire for heart failure symptoms ^, and the Seattle Angina Questionnaire ^, to quantify angina severity.

In cardiothoracic surgery, many clinical trials and research studies now include patient-reported outcomes. ^, Finally, an STS Taskforce is examining approaches to routinely collect short- and long-term patient-reported outcomes, including issues such as collection mode (email, smart portable electronic devices, standard mail) and optimal timing, frequency, and duration of follow-up.

The other major type of patient-centered outcome is patient experience of care . This should be differentiated from patient satisfaction , which is more subjective and highly dependent on whether expectations were met. ^, Experience of care instruments use structured, standardized, objective domains such as the hospital environment or discharge coordination. ^, Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) scores, developed by the AHRQ and used extensively by CMS in many of its quality programs, are the most well-known examples. In contrast to many online rating services, HCAHPS scores are methodologically rigorous and validated. HCAHPS surveys include information on doctor and nurse communication, medication communication, staff responsiveness, discharge information, care transitions, hospital cleanliness and quietness, overall hospital rating, and willingness to recommend. They adjust for patient characteristics and mode of administration. Importantly, patient experience of care scores have been associated with better adherence to treatment plans and improved outcomes for patients, ^, as well as advantages for providers including practice loyalty ^, and lower malpractice claims. ^, In addition, the skills necessary to improve patient experience scores can be taught relatively easily and quickly to physicians and their staff.

Failure to rescue quantifies mortality for patients who experience a serious complication following treatment (typically a surgical procedure). Formally stated, it is the conditional probability of death given the occurrence of a serious postoperative complication. Unlike mortality and complications, which are significantly associated with patient severity of illness, failure to rescue is more strongly associated with hospital characteristics and resources (e.g., teaching intensity, volume, nurse/patient ratios, technology services, 24/7 intensivists, closed staff intensive care units, rapid response teams, and the use of hospitalists) than with patient comorbidities. Hospitals may have more complications because they routinely care for higher acuity patients, but their resources, experience, and expertise allow them to salvage these patients and prevent serious complications from leading to death.

In cardiac surgery, Edwards and colleagues demonstrated an overall failure to rescue of 22.3% for renal failure, 16.4% for stroke, 12.4% for reoperation, 12.1% for prolonged ventilation, and 10.5% for the composite outcome. Mortality increased with certain combinations of complications and number of complications. Expanding on these findings, Kurlansky and colleagues found that for patients undergoing CABG, valve, or combined procedures, there was wide variation in failure to rescue across centers; rigorous risk models were developed, and among STS Database participants, 5.6% performed worse than expected and 4.7% better than expected.

Rarely does a single measure encompass all important aspects of a complex construct such as intelligence, stock market performance, or healthcare provider quality. In such situations, a commonly used approach is to create a composite measure that aggregates the results of multiple domains into a single score. Composite measures also increase the number of domains and endpoints, providing greater information with which to classify and differentiate provider performance. However, there is the potential for excellent scores in one domain of the composite to obscure poor performance in other domains (see online Appendix 8A ).

In cardiac surgery, mortality was for decades the most common and often the only quality metric. However, with declining mortality, it became increasingly difficult to differentiate provider quality based on this relatively infrequent endpoint. In 2007, the STS developed its first composite quality metric for CABG, ^{, ,} which included four domains ( Fig. 8.5 and online Appendix 8A ), of which two were outcomes (risk-adjusted mortality and risk-adjusted occurrence of any of five major complications: reoperation, renal failure, stroke, sternal wound infection, or prolonged ventilation) and two were process measures (use of at least one ITA bypass graft conduit and use of all four NQF-endorsed perioperative medications: preoperative beta blockade, discharge antiplatelet agents, statins, and beta blockade). In the final composite score, the relative weights of these four domains, based on the reciprocals of their standard deviations in the development sample, were risk-adjusted mortality 81%, risk-adjusted morbidity 10%, ITA use 7%, and medication use 3%.

The prototypical Society of Thoracic Surgeons (STS) coronary artery bypass grafting (CABG) composite measure contains two outcome domains (risk adjusted mortality and risk-adjusted avoidance of all five major complications) and two process domains (use of at least one internal mammary [thoracic] artery graft, and use of all four National Quality Forum (NQF)-endorsed perioperative medications). Outliers are classified using a Bayesian hierarchical framework with 95% credible intervals, corresponding to 97.5% true Bayesian probabilities. ASA, acetylsalicylic acid; preop, preoperative.

This composite provides greater dimensionality of quality measurement and acknowledges the importance to patients of nonfatal but serious complications, which may have life-altering consequences. Because of the increased information in the four composite domains, the ability to differentiate high and low outliers from “as expected” performance is greatly enhanced. In the original development cohort, only 1% of STS participants could be identified as high or low outliers based on mortality alone, but 23% of outliers were identified with the composite. In 2022, STS updated this CABG composite to include 3 years of data and to use 95% rather than 98% credible intervals, which further increased the ability to differentiate outliers. STS has now created a family of composite performance measures, all of which consist of at least risk-adjusted mortality and morbidity domains, and these encompass all the most common procedures performed in adult cardiac surgery. ^{, , ,}

The quality of a healthcare provider (surgeon, hospital, system) is typically estimated from periodically repeated samples, often consisting of relatively few cases (e.g., 50–200). These samples are used to compare the provider’s results with those of the benchmark population (e.g., US national data from all STS participants). Sample collection periods typically range in duration from 1- to 3-years, and in multiyear periods the results are updated annually by each new year’s data.

Thus, most provider performance measures are snapshots in time that serve as estimates of the true, underlying performance. How confident can a prospective patient or regulator be that these limited snapshots in time reflect a provider’s true underlying performance, how their performance compares with that of other providers (performance classification), and whether these short-term results will be consistent, reliable, and reproducible over time?

Two specific sample sizes are relevant to the credibility of provider performance measurement—the number of patients sampled for each provider, and the number of providers. The larger the sample size of patients available for a specific provider, the more certain we can be of their true underlying performance, and the less likely their results are subject to random sampling variation. Conversely, small patient sample sizes and “low information” remain one of the most persistent, troubling limitations of most healthcare performance measurement initiatives, compromising their accuracy and reliability.

Several potential approaches are available to increase the sample size of patients for provider profiling: (1) longer data collection windows (e.g., 5 years instead of 1 or 3 years), although the relevance of the older data might be questioned; (2) more heterogeneous categories of procedures and conditions (e.g., combining multiple types of cardiac procedures), although this would be less useful for patients interested in their specific procedure and might also obscure poor results in one specific procedure type; and (3) focusing on higher levels of attribution (e.g., hospital or system measurement rather than individual physicians). Additional statistical approaches include shrinkage estimation (included within hierarchical models), which shifts results from low-volume providers (which can often be more extreme) toward the mean for all providers, introducing bias to effectively anticipate regression to the mean, and producing estimates with less total mean squared error. ^{, , , , , , , ,}

Regarding the number of providers, the larger the population of providers available to study, the more generalizable will the results be and the less likely that a few extreme providers (e.g., with very high mortality) could have a disproportionate effect on the benchmark populations used to develop risk models. The latter scenario could result in models that effectively obscure identification of a true low-performing outlier. Normand and Shahian discuss replication approaches to mitigate this problem.

When presenting point estimates of a provider’s outcomes, it is recommended that they also include a measure of statistical uncertainty, such as frequentist confidence intervals or Bayesian credible intervals, both of which are inversely related to the sample size. From a frequentist perspective, if a provider’s outcomes were estimated repeatedly in different random samples of patients, the true outcome value would be contained by the 95% confidence interval in approximately 95 out of 100 samples (for any single test interval, the true value either does or does not lie within the interval). From a Bayesian perspective, the interpretation is simpler and more intuitive: a 95% credible interval implies a 95% probability that the true parameter lies within that interval.

For illustration, Fig. 8.6 shows the upper and lower limits produced by an exact binomial confidence interval when sample size varies from 50 to 900 and the observed adverse event occurrence (e.g., mortality) is 2%. At the sample sizes typically encountered for any specific procedures at most hospitals (e.g., <300 per year), event prevalence estimates have a wide range of random sampling error.

95% confidence intervals of a binomial proportion with a 2% event rate. There is considerable random sampling variation at typical provider volumes for many cardiac surgical procedures. At sample sizes of 100, for example, the 95% confidence interval extends from nearly zero to about 7%.

A fundamental question in quality assessment is whether a participant’s estimated performance in any given period of observation is an accurate reflection of the participant’s true performance (i.e., the underlying probabilities that generated the participant’s data). One approach to answering that question is provided by measure reliability , which may be thought of as consistency or reproducibility of scores. The reliability of a test result is often viewed from the narrow perspective of test-retest (subsequent results using the same rater or test) or inter-rater consistency (agreement among observers or tests). However, viewed more broadly, the reliability of a measure used for provider profiling reflects the extent to which the sample outcome reflects true signal about the provider’s performance versus noise , or random statistical fluctuation.

Any single performance estimate is subject to random sampling error and estimates from subsequent samples may yield different results. This random variation may impose constraints on the accuracy of any individual performance estimate. From a practical perspective, the design of a provider profiling initiative should seek to optimize reliability, and isolated extreme values should be viewed with caution, as regression to the mean in subsequent observations is likely. Most importantly, higher reliability measures are less likely to lead to provider misclassification.

As described by John Adams in his Rand monograph, reliability is “the proportion of the variability in measured performance that can be explained by real differences in performance.” In simple cases, this may be represented as between-provider variance divided by total variance, where total variance equals between-provider variance plus noise, or provider-specific variance. Reliability ranges from 0, where all measure variability is due to measurement error, to 1, where all measure variability is due to actual performance differences.

Adams notes that reliability can be calculated as “the squared correlation between a measurement and the true value,” although estimation of the latter may be challenging. For example, in calculating the reliability of its performance measures, STS estimates the unobservable “true value” (e.g., the true short-term mortality outcomes of a cardiac center) using Markov chain Monte Carlo approaches.

Larger sample sizes (caseloads), higher frequency of event occurrences (e.g., morbidity vs. mortality) and greater between-provider differences in outcomes typically result in higher reliability. Accruing data across multiple years will also increase sample size and measurement reliability, at the risk of using some older data that may be less relevant. Higher levels of provider attribution (system or hospital vs. physician-level) are inevitably associated with higher reliabilities. Composite measures include multiple endpoints and thus are typically associated with higher reliabilities than single outcomes.

Given these considerations, from a practical perspective surgical procedures of interest for outcomes measurement should have a relatively large number of total cases, reasonable end-point frequency (e.g., mortality is not a good end point for bariatric surgery, as it is so low), and demonstrable variation across providers in the population of interest. Further, level of provider attribution should also be selected to maximize reliability, which is typically much better at the hospital level rather than for individual physicians or surgeons. However, in cardiac surgery, using 3-year composite mortality and morbidity data for multiple common adult cardiac procedures, Shahian and colleagues developed a sophisticated, multiprocedure composite measure for individual physicians with an overall reliability of 0.81.

In cardiothoracic surgery, STS requires average measure reliability of >0.5 as a prerequisite for issuing a quality rating for its composite performance measures. Included with its more recent measure publications ^, are extensive tables showing average reliabilities for programs above and below various volume thresholds. Table 8.1 is a reliability table from Shahian and colleagues’ publication of the STS multiprocedural composite measure. These tables are used to define lower limits on programmatic volumes that would make them ineligible to receive the STS measure ratings.

Even in settings with high reliability, it is important to note that changes in a provider’s true, underlying performance can occur. For example, improvement could result from staffing changes or implementation of a new quality improvement intervention. Because there is often a several-year gap between patient outcomes and the release of public report cards based on those outcomes, when interpreting such data it is essential to consider both the statistical reliability of the performance estimate and the possibility of changes in true performance. A host of potential sources of bias may further complicate assessment of performance as described by Kahneman in Thinking Fast and Slow and by Reason in Human Error (see “ Human Error ” in Chapter 7 ). ^,

Another ramification of sample size is statistical power. One of the main functions of healthcare provider profiling is to detect providers with outcomes that are statistically better or worse than expected, as discussed in the next section on rating and outliers . In the context of statistical hypothesis testing terminology, the goal is to avoid type II statistical errors (false negative, or failure to reject a null hypothesis that is actually false). In healthcare provider profiling, this means failure to recognize that a provider’s results are significantly different from what would have been expected for their case mix.

Using a typical alpha level of 0.05, many statistical analyses for research purposes strive for sample sizes that will achieve a power of 0.80, meaning that of 100 true outliers, 80 would be correctly classified as such. Statistical power is affected by the number of procedures and the expected number of occurrences (e.g., deaths) in the time period of interest (e.g., 1, 3, or 5 years). For many of even the most common surgical procedures, unrealistically large sample sizes would often be required to achieve this power, as demonstrated in a UK study by Walker and colleagues. With national mortality of 2.7% for cardiac surgery, 352 procedures would be required to achieve a power of 0.8 and 192 to achieve 0.60 power, whereas median annual procedures per surgeon are 128. For procedures like esophagectomy or gastrectomy with substantially higher mortality, far fewer cases are required to achieve statistical power in the 0.60 to 0.80 range. Similar findings were previously noted by Dimick and colleagues.

As introduced in the previous section, one of the most common applications of healthcare quality measurement is the classification of performance categories. This is typically done using rating (e.g., “better or worse than expected” or “as expected”) or ranking (i.e., #1, #2,…), the latter mentioned only to discourage its use. As noted in numerous statistical studies, the confidence intervals around rank estimates are invariably wide and overlapping, which makes most rank order estimates both unwarranted and misleading. Further, these effectively represent a series of program-to-program comparisons that would necessitate adjustments for multiple comparisons to mitigate type 1 statistical errors. Such adjustments are rarely, if ever, performed.

Performance rating systems typically use one of two approaches to classify outliers. In criterion-based approaches, thresholds for acceptable and unacceptable performance are established based on previous literature, policy goals, or other considerations, but not specifically on the distribution of results in the current data sample. This approach is rarely used in contemporary health care, mainly because it would often be challenging to achieve broad consensus on where to set these thresholds.

Norm-based thresholds use the actual distribution of outcomes in the population of providers being assessed to determine quality outliers. There are two major norm-based approaches— partitioning and statistical hypothesis testing . In partitioning , the distribution of provider scores is divided into quantiles (e.g., deciles, quartiles, quintiles). Typically, the highest and lowest quantiles are assigned as outliers, which effectively predetermines a certain number of providers will be thus classified. However, depending on the shape of the score distribution in the provider population (e.g., narrow vs. wider), even those providers in extreme quantiles may have outcomes that are not statistically significantly different than expected. In the alternative and preferred approach, statistical hypothesis testing , appropriate tests are conducted to determine, with a specified degree of certainty, whether a provider’s results are statistically significantly better or worse than expected for their case mix.

In cardiac surgery, Shahian and colleagues used 2009 data from the STS Adult Cardiac Surgery Database to compare outliers determined from statistical hypothesis testing ( P <.05) compared with partitioning into quintiles. Among 972 programs studied, 195 and 194 programs, respectively, were in the first and fifth quintiles of risk-adjusted CABG mortality and could be considered outliers using a partitioning approach. Of these, only 8.7% of programs in the best-performing quintile and 28.4% in the worst-performing quintile were true statistical outliers, illustrating how partitioning approaches have the potential to misclassify outliers (generally exaggerating their numbers) whose results are not statistically significantly different from expected.

Risk adjustment is generally appropriate when there are patient factors that (1) are present before interaction with the provider, (2) are beyond the locus of control of the provider, (3) have an independent impact on outcomes, and (4) vary in frequency across providers. Even a high-risk predictor such as cardiogenic shock may not need to be included in risk models if all providers’ prevalences of patients with this risk were identical.

From a practical perspective, risk adjustment is also essential to ensure face validity—provider acceptance and belief in the accuracy and fairness of quality measurement. In its absence, providers who receive a low score will simply argue that the severity, acuity, or complexity of their patients were not accounted for, and some providers may consequently decline to accept high-risk patients. ^,

Most risk models in health care, including cardiac surgery, are estimated using conventional logistic regression or a multilevel hierarchical version of logistic regression. The latter approach, introduced into provider profiling in the late 1990s, specifically acknowledges and is designed to mitigate a number of issues with standard logistic regression. These include the variable and often small sample sizes from individual providers, with resulting variation in the precision with which provider performance is estimated; clustering of observations within providers; failure to appropriately partition random and systematic variation; and failure to account for multiple comparisons when estimating the results of numerous providers. ^{, , , ,} Nonhierarchical approaches that do not account for these issues can underestimate the random component of interprovider variation and, depending on context, have greater potential for false outlier classification. Hierarchical models typically produce somewhat more conservative estimates of the number of statistical outliers, although the practical differences from nonhierarchical logistic regression approaches have not been substantial in several large studies. ^,

Existing risk models have several limitations, including absence of important risk factors that are simply unavailable for many patients, so-called unmeasured confounders . However, an even more fundamental concern and source of confusion involves how these measures should be interpreted, specifically in the context of direct versus indirect standardization. ^{, ,} In direct standardization , such as age- and sex-adjusted comparisons of mortality across states, outcomes from a limited number of strata (age, sex, etc.) from each provider’s patient population are applied to a standard population. Because all providers have their outcomes applied to this same standard, these directly standardized outcomes can be directly compared with one another.

By contrast, largely because of the number of risk factors that must be accounted for (which makes stratification impractical), risk-adjusted healthcare outcomes almost always use indirect standardization , in which population-derived risk factors (typically derived from regression modeling) are applied to each provider’s patient population. A provider’s indirectly standardized outcomes can only be compared with the expected outcomes for a similar cohort of patients based on data from the overall benchmark population of providers. They should not be compared with the results of any other specific provider unless the two providers are shown to have nearly identical, overlapping risk distributions, which is rare.

A second feature of indirect standardization is that provider’s risk-adjusted scores only represent their performance for the specific mix of patients for whom they actually cared, and this case mix may vary dramatically among providers. It is inappropriate, for example, to compare an as-expected rating for a referral center caring for mostly high-risk, complex patients with a better-than-expected rating for a small center caring for mostly low-risk patients. Both institutions are doing a good job for their respective mixes of patients, but it would be unsubstantiated and inappropriate to say that the smaller program with lower-risk patients and better-than-expected outcomes is superior to the referral center with an as-expected rating.

What risk factors are permissible to use in a risk model depends on how that model will be applied. Some healthcare risk models designed for internal operational use might incorporate a broad range of variables that include preoperative patient characteristics as well as events that occur during hospitalization. For example, consider a predictive model developed to assist hospitals in identifying patients at high risk of readmission. These patients might benefit from enhanced postdischarge services to reduce their likelihood of requiring readmission. This operational model would legitimately include not only preoperative demographic and clinical features but also some events occurring postoperatively that might impact readmission risk, such as postoperative infections, stroke, or heart attack, which will impact a patient’s recovery and rehabilitation.

In contrast, when developing risk models to be used for profiling provider performance for specific procedures or conditions, only those risk factors present before a patient’s first interaction with the provider for that specific episode of care are permissible. Several categories of risk factors are typically considered for inclusion in risk models used to assess provider quality. Among these, the most common are demographic and clinical risk factors such as age and biologic sex, history of previous interventions, and comorbidities such as heart failure, low ejection fraction, renal insufficiency, or previous stroke. These are entered into multivariable risk models, and the independent association of each variable with the end point of interest is determined.

A second, increasingly studied category of risk factors includes functional status and frailty . Functional status is an individual’s ability to conduct their normal activities of daily living and to maintain an acceptable health-related quality of life. Previously described in this chapter in the context of patient-reported outcomes, identical or similar functional indicators collected prior to treatment (e.g., preoperative PROMIS score, NYHA class, Kansas City Cardiomyopathy Questionnaire, or Seattle Angina Questionnaire scores) may be associated with a variety of short- and long-term healthcare outcomes, including death, complications, and prolonged length of stay. ^{, , , ,} Some national recommendations have suggested including functional risk factors in healthcare risk modeling, as they may be important mediators of the effects of social risk factors.

Although closely related to and overlapping with functional status, frailty is a distinct phenotype characterized by reduced physiologic reserve and greater vulnerability to stressors such as illness, injury, or surgery. Single or multidimensional indicators of frailty (e.g., the Clinical Frailty Scale [CFS], Acute Frailty Network Criteria, Derby Frailty Index, Rockwood Deficit Index, Fried Frailty Phenotype, Edmonton Frailty Scale, Study of Osteoporotic Fractures, the Cardiovascular Health Study, Johns Hopkins Adjusted Clinical Group Frailty Indicator, Hospital Frailty Risk Score, and the Preoperative Frailty Index) have included various metrics such as gait speed and aerobic capacity, upper and lower extremity strength, unintentional weight loss, cognition, weakness or exhaustion, history of falls, incontinence, sarcopenia, inability to perform activities of daily living, and various lab tests, such as anemia, low albumin, and inflammatory markers. Because of its substantial impact on outcomes, preoperative identification and mitigation (e.g., prehabilitation) of frailty are subjects of increasing clinical interest.

Frail preoperative status is increasingly recommended for inclusion in risk prediction models. In cardiac surgery, various measures of frailty have been associated with increased postoperative mortality and morbidity, readmissions, worse short- and long-term overall and functional survival, and quality of life. ^{, , ,} The STS Database includes a well-validated frailty measure, the 5-meter walk test, but it has been inconsistently used by cardiac centers. New approaches to frailty assessment for STS risk modeling are currently under investigation.

The importance of a third category of risk predictors— social risk factors —has also been increasingly acknowledged. In a framework developed by the National Academy of Sciences, Engineering, and Medicine, these include socioeconomic position; race, ethnicity, and cultural context; sex and gender; social relationships; and residential and community context. Social risk factors are independently associated with healthcare outcomes, regardless of quality of care (see “ Social Determinants of Health ” in Chapter 7 ). Multiple mechanisms may be involved, including employment; income; housing; food security; preferred language, education, and literacy; availability of public transportation; neighborhood safety and recreational opportunities; community and social support systems and health resources; insurance; provider availability; residual clinical comorbidity unaccounted for by standard clinical risk factors; functional disability and frailty; postdischarge care and follow-up; and nearby pharmacies and affordable prescriptions.

Often, these social risk indicators are collectively referred to as socioeconomic position, sociodemographic variables, or social determinants of health. These may be measured at the level of the individual (patient-specific), geographic area or neighborhood , or hospital (proportion of vulnerable patients cared for by a hospital). Individual patient-level social risk variables are generally unavailable except for insurance type, such as Medicaid or dual-eligible (Medicare and Medicaid) status. Area-based indicators are more commonly used, both as proxies for individual patient-level indicators and also because of the substantial impact that the local community has on patients’ health and healthcare outcomes (e.g., access to health care; grocery stores with healthy food options; pharmacies; social support mechanisms; public transportation; recreational spaces; community services; and good quality water and air). Historically, zip codes were the most common area-based social risk indicators, but zip code areas were designed for efficient mail delivery and are not optimal approaches to defining populations that are relatively homogeneous with respect to social risk. Newer area-based indicators include those developed by Diez-Roux and colleagues; the AHRQ sociodemographics indicator; the Distressed Communities Index; and the Area Deprivation Index, a comprehensive, overall indicator of social determinants of health that incorporates 17 variables in multiple domains.

Patients with low socioeconomic/sociodemographic status often have reduced or delayed access to care, especially to specialists, and may be cared for by lower-quality providers. They have worse healthcare outcomes across a broad spectrum of conditions and procedures, including multiple studies in cardiac surgery, some of which focused on the important social risk indicator of race. The issue of social risk factors, their impact on public reporting and payment, guidelines for their use to minimize unintended negative consequences, and alternative approaches to social risk adjustment (e.g., stratified reporting, direct payment incentives, and technical assistance for the care of vulnerable populations) have been discussed in multiple publications, including a comprehensive review by Shahian and colleagues ^{, ,} and extensive reports by the NQF; the National Academy of Sciences, Engineering, and Medicine; ^, and the Office of the Assistant Secretary for Planning and Evaluation in the US Department of Health and Human Services. ^,

Although there is considerable variation among the recommendations in these reports, there is general consensus that if such variables are included in risk models, developers should describe (1) the hypothesized conceptual model by which these variables are linked to outcomes; (2) details regarding the specific variables used to capture these social constructs; (3) empirical data demonstrating the performance of models with and without these variables and across multiple different populations, including those with increased social risk; (4) the potential impact of social risk factor inclusion or exclusion on vulnerable populations; and (5) acknowledgment that for some social risk variables such as race and ethnicity, currently captured data may be oversimplified proxies for a heterogeneous range of factors including socioeconomic deprivation, environmental exposures, lower access to high-quality providers, discrimination, and epigenetic and genetic factors. When more specific indicators become available, they should be substituted for these proxies. Whether or not social risk factors are used for adjustment, results stratified by social risk categories (including race and ethnicity) are always appropriate to identify and address disparities.

With respect to risk stratifying patients according to the operation performed, RACHS-1 included 79 different types of pediatric cardiac operations grouped into six categories ranked in order of increasing risk, as perceived by clinicians. This scheme appears to be useful as a basis for forecasting risk, has been validated in a range of contexts, and has been widely referenced. One limitation is that the entire range of operations described for congenital heart disease is not encompassed. Therefore, a substantial proportion of procedures do not have a risk category attributed to them. The scheme considers operative information only, with some minimal information on age ranges for a small number of procedures. A further limitation is that it was developed before large amounts of registry data were available and, therefore, was based on clinician expert opinion rather than empirical information.

Another consensus-based scoring system developed by clinicians to describe perceived surgical risk was the Aristotle Score. It was based on three components: perceived risk of mortality, morbidity, and perception of technical difficulty. A panel of experts made estimates of these three factors for 145 procedures. There were two versions of the Aristotle Score, the more detailed of which required collecting 248 variables. This high level of data collection was perceived as a barrier to the score being implemented, noting it was less widely used than RACHS-1.

Accrual of standardized data in large multiinstitutional databases over years led to a shift from using consensus-based risk stratification tools to risk stratification based on empirical data. In 2009, an empirical risk-adjustment tool (now referred to as the STAT score) was published using information from two large databases in North America (STS, 43,934 episodes of care) and Europe (EACTS, 33,360 episodes of care). Procedures were assigned a numeric score, calculated using a Bayesian model, and performance of this model was validated in a test set. One of its aims was to group procedures with similar estimated mortality risks into relatively homogeneous groups that could then be used to adjust for case mix when analyzing outcomes and benchmarking. The risk model performed well and compared favorably with previous schemes used for risk stratification based on consensus, such as RACHS-1 and Aristotle.

The STAT (Society of Thoracic Surgeons–European Association for Cardio-Thoracic Surgery) score of STS-CHSD has undergone extensive serial improvement over time, with addition of comorbidity information and other refinements. ^, It was revised in 2021 and designed to incorporate a revised (updated) approach to multiple component operations. Scores for multiple component operations were assigned when the statistically estimated risk of mortality was different from that of the highest-risk component procedure.

The Partial Risk Adjustment in Surgery (PRAiS) model of the NCHDA, developed and validated in the United Kingdom in 2012, incorporated information about procedure (29 categories), cardiac diagnosis (3 risk categories), number of functioning ventricles, age category (neonate, infant, child), age and weight as continuous variables, presence of a non–Down syndrome comorbidity, and whether surgery was performed on cardiopulmonary bypass. As it began to be used to monitor outcomes and report these in public in the United Kingdom from 2012 onward, data quality for information previously collected but not actively used (e.g., comorbidity and diagnosis codes) rapidly improved. Moreover, as in the STS-CHSD, contemporaneous improvements were noted in 30-day mortality in the United Kingdom. These evolutions motivated an update to the PRAiS model in 2017, at which time a clinical steering panel added value to the empirical evidence to define the comorbidity and additional risk factors and the final broad cardiac procedure and diagnosis groupings used. Close involvement of the expert panel representing many hospitals and different specialties allowed careful consideration of how individual codes have been and will be used by centers in practice and built trust in the final model within the clinical community. ^, PRAiS was updated a third time in 2024, again in partnership with clinicians.

The main considerations in judging or comparing the attributes of different risk stratification schemes and risk models that have been used in pediatric cardiac surgery are as follows:

• 1)

  Coverage: proportion of cases that can be incorporated into the scheme or model, generally expressed as the percentage of cases in which a risk score could be attributed.

• 2)

  Discrimination: extent to which the scheme or model is successful in distinguishing groups of patients with higher and lower postoperative mortality. The standard measure of discrimination adopted in this literature is the area under the receiver operating characteristic (ROC), area under the curve (AUC), or C statistic. The AUC gives the probability that a patient who died, chosen at random, would have had a higher risk score than a randomly selected survivor (see alternative under “Rare Occurrence of Postoperative Events” in online Appendix 8B and “ Discrimination ” and “ Imbalanced Data ” in Section IV of Chapter 7 ).

• 3)

  Calibration: extent to which magnitude of calculated probabilities tracks relative frequency of observed outcomes. For models that estimate a percentage risk of postoperative death, a common measure of model calibration performance is the Hosmer-Lemeshow chi-squared statistic (see “ Calibration ” in Section IV of Chapter 7 ), which indicates how statistically significant any deviations are between predicted and observed mortality in groups of operations, defined by deciles of predicted risk. More recent methods include calibration slope and intercept.

To develop and test PRAiS in the United Kingdom, mean adjusted deaths compared against predicted (MADCAP), a cumulative sum (CUSUM)-type empirical graphical method for assessing model performance was used to assess accuracy of candidate models under evaluation. MADCAP enables the visual identification of patterns of systematic over- or underestimation of risk using a risk model. ^,

Common mortality end points are death in hospital or death within 30 days of the operation. Consistent verification of outcomes outside the hospital environment is not available in many databases. One problem with limiting the end point to death in hospital is that patients may die shortly after discharge home, and a subset of deaths may be related to the operation. If the end point is limited to a 30-day cutoff, although a consistent end point, patients have extremely variable hospital lengths of stay, with outlying patients staying longer than 30 days having a greater mortality risk than those with shorter lengths of stay. At the same time, use of hospital discharge (or eventual death) as the end point for such analyses may mean that children with a very long stay do not contribute for a prolonged period, also a drawback. Moreover, patients may be discharged from one institution to another, sometimes still on life support, some of whom may die at the second institution; these outcomes could be wrongly excluded from the analysis. Discharge practices might also vary between centers, which could bias a measure using only in-hospital mortality.

Morbidity associated with pediatric cardiac surgery refers to illness or lack of health that has a temporal connection to such an operation and, as such, may be regarded as a complicator or an adverse outcome. The STS Taskforce Subcommittee on Patient Safety in the United States has defined a range of adverse events that contribute to postoperative morbidity, including complications, adverse events, harm, medical error or injury, and near misses. This Taskforce further noted that in the current era of declining mortality after pediatric cardiac surgery, improvement in health care as measured by reduction in adverse outcomes is more likely when unwanted events are acknowledged, measured, and responded to in terms of healthcare delivery.

The key feature of real-time monitoring is trend rather than overall numbers. Trends are an early indicator that a process may be changing in your system, whether it’s getting better or getting worse. If a policy change has been instituted (e.g., a new treatment protocol or ward reorganization), looking at outcomes over time allows any potential changes to be tracked in real time. The most common way to do this trend monitoring is by simply plotting relevant outcomes over time, called a run chart . ^, There are many ways to display run charts, but a time series is the core element. For congenital heart disease outcomes, some form of risk adjustment should be incorporated to aid interpretation, given the heterogeneity of conditions and risk.

In the United States, PC4 facilitates this local monitoring through dashboards available to participating centers. The dashboards include run charts of outcomes for each center, adjusted for case mix, and allow centers to stratify by patient characteristics and drill down into individual patient records if required. The design of the dashboard is explicitly tailored to enable quality improvement, with the innovation that participating centers can see the results of other centers and identify where there can be mutual learning. Evidence supports that this approach contributes importantly to improving outcomes for participating centers.

In the United Kingdom, the National Health Service has mandated that all centers performing pediatric congenital heart surgery must examine their outcomes at least once a month using a type of run chart called a variable life-adjusted display (VLAD), ^, which charts the cumulative difference between expected and actual risk-adjusted mortality over time. Spiegelhalter has provided seven important aspects that need to be considered when designing monitoring systems: (1) Unit being monitored; (2) Event being monitored; (3) Risk adjustment of outcomes; (4) Choice of summary statistic for monitoring; (5) Role of discounting past experience; (6) Selection of thresholds for action; and (7) Actions to take.

Spreadsheet software has been developed for clinical teams to enable regular monitoring of unit-specific outcomes using VLAD charts. The expected risk of death is estimated for each 30-day episode of care using PRAiS, and risk-adjusted outcomes over time are displayed using the VLAD method. Additional cardiac operations and interventional catheterization within each 30-day episode of care are also displayed on the VLAD chart to provide additional context on achieved outcomes.

The software also provides additional information to support learning. Individual records for all patients who died are collated on a worksheet, along with calculated risk factors; estimates of risk for each patient are available, as are all risk factors (e.g., diagnostic groups or comorbidity group assignment) used in the PRAiS risk model. This allows for sense checking and subgroup analysis within local teams (e.g., all univentricular heart patients). Finally, the software provides information on how a particular unit’s risk factors compare to the national cohort and provides (user-specified) prediction intervals for overall observed 30-day mortality.

A crucial aspect of the software is that it was co-designed with clinical teams. At each time the PRAiS risk model is updated, further user input is sought on the software to incorporate additional improvements. This principle led to addition of functions, including comprehensive error checking of the raw input data that helps local teams maintain data quality and the ordering and design of columns to enable rapid data transfer. Such codevelopment supports implementation and sustained use: the PRAiS software has been in consistent use within United Kingdom centers since 2013, with minimal technical support.

In the United States, PC4 also provides data on a range of complications, such as cardiac arrest, mechanical circulatory support, unplanned cardiac reintervention, neurologic complications, chylothorax, and failed extubation. ^{, ,} Occurrence of these complications varies greatly across participating centers (e.g., a fivefold difference in chylothorax), and the PC4 dashboard facilitates intercenter learning through access to unblinded results and a consortium commitment supporting teams from one hospital learning from another.

In the United Kingdom, the National Institute for Cardiovascular Outcomes has started collecting data for some complications after pediatric cardiac surgery, and unadjusted center-specific prevalences are reported annually; however, the time interval between occurrence and report of over a year is an important disadvantage. Prototype methods for more contemporaneous feedback of recent center outcomes on a range of complications, such as cardiac arrest, mechanical circulatory support, unplanned cardiac reintervention, neurologic complications, chylothorax, postoperative bleeding, and feeding problems, have been developed together with clinical teams. Results are presented in a series of slides automatically generated from a spreadsheet, providing frequency of each complication, frequency of multiple complications in a single patient, and, for any given complication, which other complications have been coincident. These outcomes are currently presented as unadjusted prevalences benchmarked against the national average, with plans to add risk adjustment. New models for complications after surgery for congenital heart disease in England and Wales have been constructed for 6 morbidities occurring within 30 days of adult congenital heart surgery and for seven morbidities within 30 days of pediatric congenital heart surgery.

Traditionally, local improvement initiatives have focused on short-term outcomes that can be measured and reported in close to real time (typically with no more than a 30-day lag), to allow reflection and improvement in acute hospital management. However, a more holistic viewpoint is the long-term outcome for a child born with a given diagnosis, particularly survival. This is what matters most to families, but also to commissioners, as the ultimate aim of a program. The Congenital Heart Audit: Measuring Progress In Outcomes Nationally (CHAMPION) project is exploring ways of including 5-year survival and information on off-pathway procedures as part of annual reports as a way of moving toward long-term outcomes reporting. However, it must be acknowledged that long-term outcomes are less useful for ongoing improvement initiatives because often the important events occur long before they are reported. That said, they do provide an important complementary perspective and are often the most useful outcomes for evaluating different treatment options, for example, timing and sequencing of surgery or subtle effects of case volume. ^, The authors have proposed methods for the routine reporting of long term outcomes in the United Kingdom by diagnosis, which includes defining clear treatment pathways for the different diagnoses.

Transparency requires hospitals and healthcare systems to commit to making outcome data readily available to families, with clear explanations of what the data are, where they have come from, and how their accuracy is ensured. Accessibility is a necessary condition for transparency:

Data are not transparent if they are not accessible. There are two aspects to accessibility. The first is making it easy to find results both online and in person from a child’s physician or hospital. A PDF file buried at the bottom of a hard-to-navigate technical website is not accessible. The second aspect is comprehensibility. Jargon must be avoided, and all concepts necessary for interpreting the data must be explained. To ensure this, working closely with parents and the public on language, content, and level of detail is crucial. Pagel and colleagues codeveloped a website with parents and the public to present and explain United Kingdom monitoring of 30-day mortality after pediatric heart surgery and described how close interaction with parents was crucial at all stages. Parents highlighted jargon, confusing graphs, and key concepts to emphasize at every stage of the process, all of which had not been identified by the subject experts. Brown and colleagues followed a similar process in developing a patient information leaflet for parents explaining different complications of pediatric surgery, how common they were, and what their implication was for survival or length of hospital stay. The content was driven by workshops with parents whose children had experienced complications, and a key theme was their wish that they had known beforehand that it could happen and what it might mean.

In the United States, the patient advocacy group Conquering CHD has a Guided Questions Tool that encourages parents to ask key questions about their child’s cardiac team. In addition to questions specific to their child, they suggest a series of questions aimed at understanding the hospital’s patient survival, case volume, complications, and whether the hospital contributes to national benchmarking and publishes its results. Several hospitals have responded by making their data highly accessible and providing detailed outcome data benchmarked to national data.

Presenting outcomes in the appropriate context is essential to avoid misinterpretation. Data must be presented alongside contextual information, such as case-mix complexity, patient demographics, and procedural volume, to help parents understand the factors influencing outcomes. Presenting risk-adjusted outcomes can be a powerful way of contextualizing the data, but in this case, risk adjustment must be explained, and its caveats must be transparently presented. The UK website “Understanding Children’s Heart Surgery Outcomes” has layers of detail about what risk adjustment is and how it is used, and a section on what the site cannot be used for ( Figs. 8.7 , 8.8 , and 8.9 ).

Layering information so that families can choose the level of detail they access can also enhance interpretability and transparency. The amount of information provided by clinicians can have both positive and negative effects on parental anxiety. ^, Parents want information to be accessible, honest, and understandable in lay language. Because each family is different, with some wanting to know every detail and others wanting less, layering information helps create resources that families can tailor to their own needs.

Layering information is also important when considering that many families access outcome data at a time of stress shortly after their child has received a congenital heart disease diagnosis and, often, shortly before treatment must begin. Stress and time pressure are known to be barriers to informed consent. ^, Studies have also indicated that distress levels, time pressure, and the quality of “clinician–parent communication” can adversely affect parental understanding of information provided, ^, with the potential to limit parents’ ability to make informed decisions. Thus, information on outcomes should not only be layered but repeated in several places throughout a document or on a website, with key concepts highlighted to draw the eye and short, easily understood sentences. The UK website developed such a “key points” section following the direct suggestions of families.

Understandably, what most concerns parents is what is likely to happen to their child, such that the risk of death or complications would ideally be tailored to the individual characteristics of the child. This is something that national or center-level outcome reporting cannot do, and this limitation must be made clear. Such conversations are most appropriate between a family and their child’s direct clinical team, who know the child so much better than a relatively small set of risk factors can convey. A clinical team must also be careful to support parents and families in coping with uncertainty.

In an era of Evidence-Based Medicine, to our knowledge, no randomized trial of the value of public reporting has been tested, and only observational data are generally cited without accounting for confounders. However, as is evident in the preceding text, in the United Kingdom there is a unique center: the Winton Centre for Risk and Evidence Communication based in the Department of Pure Mathematics and Mathematical Statistics at Cambridge University whose chair is Sir David Spiegelhalter. He, with Drs. Christina Pagel and Katherine Brown in the Clinical Operational Research Unit at University College, London, and others in psychology and neuroscience at King’s College, London, launched a three-round workshop to investigate how well the public, parents, families, and healthcare workers understood what was being reported to the public and if it could be improved. Here are the words of Sir David as he reflected on this seminal research:

“ This has been a humbling and invaluable experience. I thought I knew something about communicating statistics, but sitting listening to enthusiastic users struggling to understand concepts made me realize my inadequacy. If we want to genuinely communicate statistical evidence, I am now utterly convinced that users have to be involved from the very start. ”

Here are some important findings:

• Risk ratios are hard to understand. This is particularly problematic when risk is very low. For example, 10% risk is twice that of 5% risk, but how important is the twofold difference between.05% risk and.1% risk or.005% risk and.01% risk? As noted in Chapter 7 , such ratios represent what statisticians call imbalanced data. In such cases, risk models are guaranteed to have a high area under the receiver-operator curve (AUC or C-statistic). Better statistics are the area under the precision-recall curve and G-mean, among other statistics for rare events.

• Parents, families, and we suspect adult patients grasp survival (being alive) better than mortality. Thus, the idea that 99 out of 100 patients will leave the hospital alive is better understood than 1% mortality. This resulted in tables and graphs of survival rather than of mortality ( Fig. 8.8 ).

  

  Screenshot of survival data for Great Orman Street Hospital for Children from the public website https://www.childrensheartsurgery.info/data/map . Note that survival, not mortality, is reported, that information is provided in easily readable format, and an explanation is given in language that has been vetted by parents and psychologists.

• Horizontal bar graphs are superior to vertical ones. Newspapers figured this out long before these experiments, with the majority of “bar graphs” displayed horizontally ( Figs. 8.8 and 8.9 ).

  

  Background, limitations, and a simple verbal explanation assist in educating parents and the public in understanding the outcome statistics. Only survival statistics are shown because surveys found that this was far and away the most important concern. ( https://www.childrensheartsurgery.info/faqs ).

• Some metric on where a given patient, hospital, or provider risk sits in the range of expected survival is helpful ( Figs. 8.8 and 8.9 ).

• Words matter. The authors noted: The experiments and workshops also emphasized the importance of consistency in using (or implying) terms such as “luck,” “chance,” “risk,” “uncertainty,” and “probability.” We decided always to refer to predicted risk as “predicted chance of survival.”

• Simple education is key, both by simple short explanations and by “cartoons” ( Fig. 8.7 ).

  

  Simple cartoons preceed a more thorough and extensive explanation of congenital hearat surgery survival statistics the parents and public are seeing. ( https://www.childrensheartsurgery.info/intro ).

Although there is much common ground between parents, CH caregivers, and health professionals, all of whom want to see the best outcome for the child, the range of focus for parents and caregivers differs somewhat from that of clinicians. For example, Pagel and colleagues in a prospective study to collect data on complications following pediatric heart surgery ran a rigorous and extensive outcome selection process together with clinicians and families. Some divergence was seen between the two groups regarding the fundamental issues of what the important morbidities linked to pediatric cardiac surgery are: clinicians tended to prioritize clearly clinical issues related to the heart (use of extracorporeal life support and reoperation), whereas parents placed greater emphasis on holistic outcomes for their child (problems feeding, child development, and communication). During the consent process and the postoperative periods, the highly skilled practitioners involved in pediatric cardiac surgery understandably have a particularly intense focus on the conduct and success of the procedure itself. Although parents and caregivers obviously share this priority, given their role as primary caregivers, they wish to know more about what their child will be doing afterward in the medium and longer term. Clearly, both areas are very important.

Does a performance metric measure what is important to the patient? Schaff and Bailey cautioned that something as relatively simple as the composite score– based STS ratings, which include morbid events, may not reflect the relative importance placed on various adverse outcomes by different patients. For example, some patients may fear a stroke as much or more than dying, while others may have similar fears regarding the need for lifelong kidney dialysis. In shared decision-making, it is essential that the provider elicit from the patient what their greatest priorities and fears are, which may not be reflected in the design of the applicable quality metrics for their procedure.

This procedure-related focus estimates the patient’s expected risk of an intervention, which informs shared decision-making and alerts the surgical team regarding potential challenges and mitigation strategies. Importantly, surgical procedures have an unequivocal time zero and exact denominator, and the typical performance metrics require only short-term follow-up and are often actionable. They include not only adverse events but also quantifiable process metrics, such as antibiotic timing or use of an ITA for CABG. These metrics are easily used in program rating and ranking by various professional organizations and performance rating services.

Patient autonomy and shared decision-making are also affected by the way in which information is presented to them, sometimes referred to as the “priming effect”. When patients are informed (“primed”) that they have a 10% chance of dying after a procedure, more patients decline the procedure than when they are told there is a 90% chance of surviving, despite these statements being mathematically equivalent (see Figs. 8.7 , 8.8 , and 8.9 ). ^{, ,} Similarly, physicians remember their last bad outcome much longer than their numerous intervening successes.

Costs of care and the financial viability of their hospital or health system are additional factors that may influence physician decision-making. Controlling healthcare costs is unquestionably a critical national priority. The United States spends more per capita on health care than any other developed country without concomitant superiority in health outcomes. Various macroeconomic policies have been implemented to control costs, many of which are monitored with performance indicators. The costs of specific equipment and drugs, length of hospital stay, and readmissions are a few examples of “quality metrics” that may affect care decisions without rigorous evidence to show they ultimately improve outcomes or reduce overall costs.

Consider this example. If outcomes are similar and the cost of a percutaneous valve is more than 10 times that of a surgical prosthesis, who should be responsible for the additional costs of a transcatheter procedure? Is the higher cost justified from the perspective of quality, patient benefit, and the viability of individual healthcare organizations?

From the patient’s perspective, the rational decision is clear. For a patient with severe aortic stenosis, they are eight times more likely to be alive one year later with an intervention. Transcatheter AVR appears to be associated with the same or lower risk than surgical valve replacement and avoids the months needed for full functional recovery. However, the procedural costs are higher and the margins lower for the hospital. In this context, should it be the surgeon, patient, payor, or healthcare organization that determines which procedure to perform?

As noted previously in this section and in Section I, quality and safety measures based on procedural outcomes have inherent unintended consequences including “gaming” and physician “risk aversion.” Regarding the latter, in an era of generally excellent cardiac surgical outcomes, some have argued that programs may avoid some very high risk patients (often those who may benefit most from an intervention) in hopes of preserving or improving their personal or organizational performance metrics. Others have countered that what is typically classified as the inappropriate practice of risk aversion may in some instances be an effective, patient-centric strategy, intentionally allowing higher risk patients to be transferred from a less experienced and more “risk averse” center to one with greater experience and expertise in caring for such patients.

Although little evidence exists for risk aversion in cardiac surgery, Wasfy and colleagues noted that 79% of interventional cardiologists in the State of New York thought public reporting played a role in their clinical decision-making regarding PCIs.