Background
One of the main challenges for imaging laboratories is demonstrating the quality of their studies. The aim of this study was to determine if echocardiographic training and experience are associated with the accuracy of left ventricular ejection fraction (LVEF) reporting using all-cause mortality as the gold standard.
Methods
Survival was determined for consecutive patients undergoing echocardiography at one of four academic facilities. The relationship between LVEF and survival was determined for different groups of physician readers and sonographers on the basis of board certification and experience. Studies of physicians reading <200 studies were excluded.
Results
Data from 63,108 patients and 40 physicians were included. There was moderate variation across physicians in the relationship between LVEF and 1-year mortality (area under the receiver operating characteristic curve interquartile range, 0.56–0.64). The relationship between LVEF and 1-year mortality was stronger for physicians board certified in echocardiography (area under the receiver operating characteristic curve, 0.60; 95% confidence interval, 0.59–0.61) compared with those not certified (area under the receiver operating characteristic curve, 0.56; 95% confidence interval, 0.55–0.57; P < .0001). Physician experience, years since training, and sonographer experience and certification were not clearly associated with the predictive value of LVEF. After adjustment for patient characteristics, the LVEF-mortality association of board-certified physicians remained stronger than the LVEF-mortality association of those not certified.
Conclusions
LVEF as determined by physicians board certified in echocardiography was associated with a stronger relationship with mortality than as determined by those not certified. The LVEF-mortality relationship may be useful as one measure of the quality of imaging.
One of the main challenges for imaging laboratories is demonstrating the quality of their studies. The American Society of Echocardiography recommends that quality of imaging should address interpretation in addition to patient selection, image acquisition, results communication, and incorporation of the results into care. The quality of imaging can have significant impact on patient outcomes. For example, life-prolonging therapy (e.g., β-blockers, angiotensin-converting enzyme inhibitors) is often initiated for patients with heart failure on the basis of the left ventricular ejection fraction (LVEF) from echocardiography. Although the determination of LVEF with echocardiography can be quantitative (e.g., tracing of the left ventricle), it is not automated and is dependent on the skills of the interpreter. Thus, the accuracy of the reported LVEF is a potential marker of the quality of an echocardiography laboratory.
Both board certification (physician and sonographer) and accreditation (laboratory) have been proposed as methods to determine the quality of imaging. The goal of certification and accreditation is to create a laboratory setting that produces high-quality images and reports. However, it is unclear if physicians and sonographers with more training and experience produce more accurate reports, such as those of the LVEF.
The primary objective of this study was to determine if echocardiographic training and experience are associated with the accuracy of LVEF reporting using the known association of LVEF and all-cause mortality as the gold standard.
Methods
Patients
We used data from consecutive patients undergoing echocardiography at four academic medical centers (two US Department of Veterans Affairs [VA] and two non-VA centers) located in California, Colorado, and Kansas. Patient age and gender, year of study, and LVEF were recorded. Each site was asked to provide data for ≥10,000 consecutive patients between 2000 and 2007. For patients with more than one echocardiogram, the first study was selected. The study protocol was approved by the institutional review board of each participating center. Patients were excluded if aged < 18 years.
LVEF
We used the reported LVEF regardless of its method of determination. We also calculated LVEF using the Teichholz formula, which is based on ventricular end-systolic and end-diastolic dimensions, when such measurements were available. Most laboratories reported categories (ranges) of LVEF. To standardize the measurement across laboratories, LVEF was classified in one of the following categories: ≤22%, 23% to 27%, 28% to 32%, 33% to 37%, 38% to 42%, 43% to 47%, 48% to 54% 55% to 70%, and >70%.
Physician-Echocardiographers
We determined certification by the National Board of Echocardiography (NBE; yes or no), years since completion of echocardiographic training (1–5, 6–15, of >15), and estimated number of echocardiograms read per year across all facilities (<200, 200–500, 500–2,000, or >2,000). Two physicians moved between study facilities, and these physicians’ studies from their initial facilities were excluded.
Sonographers
We recorded sonographer certification (yes or no) by the American Registry of Diagnostic Medical Sonography or Cardiovascular Credentialing International, years of sonographic experience (1–5, 6–15, or >15). One facility was not able to provide data on sonographers’ years of experience. Although at each facility, the physician determines what LVEF value to put in the report, the sonographer was responsible for measuring left ventricular dimensions. Thus, sonographers’ accuracy was evaluated using the relationship between 1-year mortality and LVEF as calculated using the Teichholz formula on the basis of the left ventricular dimensions during systole and diastole. For analyses of sonographer characteristics on accuracy of LVEF, we excluded patients if the sonographer did not perform ≥200 studies in the data set ( n = 2,005). One facility did not provide data on sonographers’ experience, and its patients were excluded for this secondary analysis.
Survival
Each site used the Social Security Death Index to determine total mortality after echocardiography. The primary outcome was survival at 1 year after echocardiography. Secondary mortality outcomes included 30-day and 5-year survival. One-year mortality was selected as the primary outcome because it balanced the need for enough events (30-day mortality may be too low) and the desire to detect an association between LVEF and outcomes quickly (5 years would be too long to wait to evaluate quality).
Statistical Analysis
Pearson’s χ 2 analysis (with Yates’s continuity correction) was used to evaluate categorical variables, and analysis of variance was used to evaluate differences in continuous variables. To determine the overall accuracy of LVEF in predicting total mortality, we used receiver operating characteristic (ROC) curves that displayed sensitivity by 1−specificity. The impact of physician and sonographer characteristics on LVEF predictive accuracy were determined by comparing the area under the ROC curves. The significance of differences in ROC areas was determined using the roccomp command in Stata (StataCorp LP, College Station, TX). For the ROC analyses, we excluded patients with LVEFs ≥ 70% ( n = 2,784), given that we observed a slight U-shaped relationship between mortality and LVEF, with higher mortality at high LVEF values.
To adjust these differences in provider characteristics for differences in patient characteristics, we used two methods. In the first, we created a logistic model of total mortality in which the patient characteristics, provider characteristics, LVEF, and interaction between provider characteristics and LVEF were included. In the logistic models, LVEF was evaluated as a categorical variable (including all LVEF values), given that we observed a U-shaped (nonlinear) relationship between LVEF and mortality. A statistically significant interaction term ( P < .05) was considered evidence of effect modification of provider characteristics on the accuracy of LVEF in predicting mortality. The logistic models were adjusted for clustering of patients at the physician level.
In the second method of controlling for patient characteristics, we evaluated the difference in the area under the ROC curves for 1-year mortality for LVEF and LVEF calculated from LV dimensions. Both areas were calculated for each provider. We determined the number of providers with similar or greater improvement in the accuracy of the final report LVEF (area under the ROC curve for LVEF report ≥ area under the ROC curve for LVEF on the basis of dimensions). Because this method already controls for LVEF internal dimensions, it should be less confounded by differences across patients. All analyses were performed using Stata version 9.0.
Results
Facility and Patient Characteristics
The patient ( n = 63,108) and facility ( n = 4) characteristics for the four participating centers are shown in Table 1 . The referral base for each facility included a tertiary care academic hospital and multiple community clinics. The mean age of the patients undergoing echocardiography varied from 56 to 66 years across the facilities. Approximately half of the patients from non-VA centers were women, while 95% of patients at VA facilities were men. A total of 40 physician-interpreters and 57 sonographers were included in the study. The standard practice at each laboratory was to report a visual estimate of LVEF that was often informed by a calculation based on a tracing of the systolic and diastolic endocardium (e.g., method of discs).
| Variable | Facility A | Facility B | Facility C | Facility D |
|---|---|---|---|---|
| Patients | 22,146 | 19,611 | 11,031 | 10,320 |
| Age (y), mean ± SD | 59 ± 16 | 56 ± 18 | 67 ± 13 | 65 ± 12 |
| Men | 49% | 52% | 95% | 95% |
| Providers ∗ | 20 | 6 | 8 | 6 |
| Sonographers ∗ | 23 | 15 | 12 | 6 |
| VA facility | No | No | Yes | Yes |
∗ Providers interpreting ≥200 studies and sonographers performing ≥200 studies.
Provider Characteristics
A majority of providers (85% [34 of 40]) were level II trained, five (12.5%) were level III trained, and one (2.5%) was level I trained. Nine physicians (23.5%) had 0 to 5 years of experience since the completion of echocardiographic training compared with 16 (40%) with 6 to 15 years of experience and 15 (37.5%) with >15 years of experience. Overall, 13 physicians (33%) were certified by the NBE in adult transthoracic echocardiography during the study period. Twelve physicians (30%) reviewed ≥2,000 studies per year, 15 (37.5%) reviewed 1,000 to 2,000, six (15%) reviewed 200 to 1,000 and seven (17.5%) reviewed ≥200 with an unclear upper limit. Two of the four echocardiography laboratories were accredited by the Intersocietal Commission for the Accreditation of Echocardiography Laboratories.
LVEF as a Predictor of Mortality
Mortality at 30 days, 1 year, and 5 years is shown in Figure 1 for patients grouped by LVEF. In general, there was an increase in mortality with decreasing LVEF. There was a slight U shape to the mortality-LVEF relationship, with a nadir in the mid-50% range of LVEF.
Physician Characteristics and LVEF as a Predictor of Mortality
The areas under the ROC curves for LVEF as a predictor of 1-year mortality are shown in Table 2 for different physician characteristics. Physicians who were board certified by the NBE had a significantly higher mean area under the curve than those not certified, but there was considerable overlap of the distributions of areas ( Figure 2 ). There were no clear differences between providers on the basis of years since training, level of training, or studies reviewed per year ( Table 2 ).
| Variable | Physicians | Patients | Area under the ROC curve ∗ | 95% confidence interval | P |
|---|---|---|---|---|---|
| Certification | <.0001 | ||||
| Yes | 13 | 32,580 | 0.60 | 0.59–0.61 | |
| No | 27 | 30,528 | 0.56 | 0.55–0.58 | |
| Studies per year | .08 | ||||
| 200–1000 | 6 | 16,671 | 0.59 | 0.58–0.60 | |
| 1,000–2,000 | 15 | 15,894 | 0.60 | 0.58–0.62 | |
| >2,000 | 12 | 26,964 | 0.58 | 0.57–0.59 | |
| >200, unknown upper limit | 7 | 4,334 | 0.64 | 0.58–0.69 | |
| Years since training | .42 | ||||
| 0–5 | 9 | 5,773 | 0.57 | 0.55–0.59 | |
| 6–15 | 16 | 21,075 | 0.59 | 0.57–0.60 | |
| >15 | 15 | 37,015 | 0.59 | 0.58–0.60 | |
| Level of training | .79 | ||||
| I | 1 | 991 | 0.60 | 0.54–0.65 | |
| II | 34 | 45,730 | 0.58 | 0.57–0.59 | |
| III | 5 | 17,142 | 0.59 | 0.57–0.60 |
A graph of the area under the ROC curve for the LVEF-mortality relationship ( Figure 3 ) reveals that board-certified readers were more accurate in their separation of normal from mildly abnormal LVEFs. Low LVEF readings had similar accuracy for those with and without board certification. The difference in board certification versus no board certification was greatest for studies without available M-mode measurements of the left ventricular dimensions (board certified: area under the ROC curve, 0.60; 95% confidence interval, 0.57–0.62; not board certified: area under the ROC curve, 0.53; 95% confidence interval, 0.51–0.55; P < .0001; Figure 3 B).
