Background
Clinical application of the American College of Cardiology Foundation Appropriate Use Criteria (AUC) represents a potentially feasible alternative to third-party pre-certification for imaging procedures and will soon be required as part of the accreditation process for imaging laboratories. Electronic tools that rapidly apply the AUC are needed in clinical practice. We developed and tested a web-based application of the AUC to track appropriateness of transthoracic echocardiography (TTE).
Methods
Indications for outpatient TTE studies performed in a university hospital echocardiography laboratory were assessed prospectively at the point of service using a prototype web-based AUC application (Echo AUC App). The Echo AUC App was developed on the basis of our own prior published data regarding indication frequency to minimize time and screens required for completion. Echo AUC App-determined indications were compared with blinded investigator-determined indications based on review of relevant medical records. Echo AUC App characteristics, including Echo AUC App entry time, were recorded.
Results
Of the 258 studies enrolled, Echo AUC App-determined TTE indications were Appropriate (A) in 77% ( n = 198), Inappropriate (I) in 9% ( n = 23), and Not Classified (NC) by the AUC in 14% ( n = 37). Agreement between Echo AUC App- and investigator-determined classifications was excellent (94%, kappa statistic 0.83). Mean Echo AUC App study entry time was 55 seconds (range 25-280 seconds).
Conclusion
The use of an electronic application allows rapid and accurate implementation of the AUC for TTE at the point of service. Such an application could be installed in echocardiography laboratories to track appropriateness in accordance with soon-to-be-implemented accreditation requirements. Further study of this Echo AUC App at the point of order may provide an alternative to third-party pre-certification procedures.
The last decade has seen a dramatic increase in the use of cardiovascular imaging. According to a review of Medicare data from 1999-2004, echocardiography accounted for approximately half of all spending on cardiac imaging services. This finding is not surprising, given that echocardiography is noninvasive, widely available, and useful in a broad range of clinical scenarios. As imaging expenditures continue to grow, physicians and echocardiography laboratories face increasing scrutiny from federal and private payers. To help providers and reimbursement agencies determine a rational approach to echocardiography use, the American College of Cardiology Foundation (ACCF) and American Society of Echocardiography (ASE) have published Appropriate Use Criteria (AUC) for transthoracic echocardiography (TTE) and transesophageal echocardiography.
Since publication, the AUC have rapidly been adopted as a useful tool to guide the appropriate use of imaging procedures. The Intersocietal Accreditation Commission has announced that tracking and reporting of appropriateness according to the AUC will soon become a requirement for cardiovascular imaging laboratories as part of the accreditation process. The AUC have also emerged as a potentially feasible alternative to the expensive and inefficient process of pre-certification for imaging procedures. Although cumbersome for all imaging modalities, pre-certification procedures are particularly impractical for TTE, given that it has the highest volume of all cardiac imaging modalities. As a result, there is great interest in exploring ways to incorporate the AUC for echocardiography into order entry systems to improve the quality and efficiency of patient care.
The application of the AUC for TTE has been studied extensively in a variety of clinical settings and has been found to be accurate, reproducible, and feasible. However, the manual processes studied to date are time-consuming and cumbersome, and practitioners not thoroughly familiar with the AUC may have to review the entire list of indications before settling on the most germane. An automated decision-tree application of the AUC is needed to allow rapid appropriateness determinations and may lead to more widespread use of the AUC in clinical practice.
In an effort to address this need, we designed a prototype web-based, automated application (Echo AUC App) that classifies the indications for TTE according to the AUC and tested the feasibility of its implementation in routine clinical practice. We hypothesized that the Echo AUC App would allow rapid and accurate clinical application of the AUC.
Materials and Methods
Outpatients aged ≥18 years who were referred to the University of Chicago Medical Center for TTEs from September 2009 to December 2009 were eligible for inclusion in this prospective study. The study was approved by the institutional review board, and all patients provided informed consent. For each echocardiogram included, basic patient demographics and pertinent clinical information necessary to determine the primary study indication were entered prospectively into a prototype web-based AUC application (Echo AUC App). Enrollment occurred at the point of service, before image acquisition. The prototype Echo AUC App is an online application of the 2007 ACCF/ASE AUC for TTE that provides a rapid decision-tree algorithm to determine indication number (1-51) and level of appropriateness (Appropriate or Inappropriate) of each study as defined by the AUC. The decision-tree algorithms in the Echo AUC App were developed on the basis of previously published AUC implementation studies at our institution, which established the frequency of studies ordered for each indication number in the AUC and defined critical branch points in the application of the AUC, to minimize the number of screens and time required to determine the appropriateness of each TTE. The Echo AUC App includes a preliminary screen that queries the user for patient demographic information (age, gender), ordering physician specialty, and absence/presence and date of prior TTE, followed by the decision-tree algorithm, with a final screen stating the resulting appropriateness determination: Appropriate, Inappropriate, or Unclassified (i.e., not addressed in the 2007 AUC for TTE).
Echo AUC App users were sonographers who were performing the transthoracic echocardiograms and physician investigators, none of whom had significant prior exposure to the Echo AUC App. Before initiation of the study, all Echo AUC App users were given a universal login and password and a brief 15-minute tutorial on the use of the Echo AUC App, including an orientation to the type of information required to complete the Echo AUC App for each TTE. All users also practiced with the Echo AUC App by entering two to five practice TTEs before study initiation. All Echo AUC App users had access to online or written study requisitions, in addition to the hospital’s electronic medical record and ordering systems. For each TTE entered in the study, Echo AUC App response characteristics were recorded, including number of screens required to arrive at a final appropriateness determination. The time to complete the application, from log-on to final appropriateness determination, was recorded automatically by the Echo AUC App for all studies. In addition, a separate physician investigator, blinded to the Echo AUC App-derived appropriateness results, manually determined the AUC indication number and appropriateness classification for each TTE from hospital records and study requisitions, using a methodology we have previously validated. Echo AUC App-derived indication numbers and appropriateness determinations were then compared with the manually derived indication numbers and appropriateness determinations in an effort to evaluate the accuracy of the Echo AUC App.
Statistical Analysis
Comparisons between Echo AUC App-derived and manually derived AUC study indications and AUC appropriateness determinations were performed with chi-square tests for categoric data and Student t test for continuous data using a two-tailed P value < .05 for statistical significance, or kappa statistics as appropriate. Interobserver variability in the determination of indication number/category and level of appropriateness (Appropriate, Inappropriate, Unclassified) is expressed as percent agreement between the Echo AUC App-derived and manually derived determinations and with the use of kappa statistics. For indication number/category determinations, interobserver comparisons represent the frequency with which the reviewers agreed on indication numbers or categories (i.e., indications 1-51, Unclassified). For level of appropriateness determinations, interobserver comparisons represent the frequency with which the reviewers selected indications with matching levels of appropriateness (A vs. I) or agreed that studies were Unclassified, although they did not necessarily choose identical indication numbers.
Results
A total of 258 TTEs were included in the study. The study and provider characteristics are shown in Table 1 . The mean age of the population was 59 ± 18 years, and 53% of TTEs ( n = 137) were performed on women. A prior TTE had been performed in 35% of cases ( n = 90). Among patients with prior TTEs, 17% ( n = 15) of prior studies had been performed within the preceding year, whereas 83% ( n = 75) had been performed more than 1 year before the index TTE. Cardiology was the most frequent referring specialty, accounting for 55% ( n = 142) of TTEs enrolled, with Internal Medicine/Primary Care the next most common (32%, n = 83). Overall, the most common indication for TTE was AUC indication 1, “Symptoms potentially due to suspected cardiac etiology…” (32% of classifiable TTEs [ n = 71 TTEs]; Table 2 ), whereas the most common inappropriate indication was AUC indication 42, “Routine (yearly) reevaluation of patients with heart failure in whom there is no change in clinical status” (5% of classifiable TTEs [ n = 11]; Table 3 ).
| No. of TTEs (%) ( N = 258) | |
|---|---|
| Female patient | 137 (53%) |
| Prior TTE | 90 (35%) |
| Within the preceding year | 15 (17%) |
| >1 year ago | 75 (83%) |
| Referring physician specialty | |
| Cardiology | 142 (55%) |
| Internal medicine/family practice | 83 (32%) |
| Other | 33 (13%) |
| Indication as listed in the AUC for echocardiography | No. of studies (%) ( N = 221 classifiable studies) |
|---|---|
| 1. Symptoms due to suspected cardiac cause, including but not limited to dyspnea, shortness of breath, lightheadedness, syncope, TIA, cerebrovascular events (Appropriate) | 71 (32%) |
| 2. Prior testing that is concerning for heart disease (e.g., chest x-ray, baseline scout images for stress echocardiogram, ECG, elevation of serum BNP) (Appropriate) | 20 (9%) |
| 17. Initial evaluation of murmur in patients for whom there is a reasonable suspicion of valvular or structural heart disease (Appropriate) | 18 (8%) |
| 10. Evaluation of known or suspected pulmonary hypertension, including evaluation of right ventricular function and estimated pulmonary artery pressure (Appropriate) | 15 (7%) |
| 42. Routine (yearly) reevaluation of patients with heart failure (systolic or diastolic) in whom there is no change in clinical status (Inappropriate) | 9 (4%) |
| Indication as listed in the AUC for echocardiography | No. of studies (%) ( N = 23 inappropriate studies) |
|---|---|
| 42. Routine (yearly) reevaluation of patients with heart failure (systolic or diastolic) in whom there is no change in clinical status | 9 (39%) |
| 21. Routine (yearly) reevaluation of an asymptomatic patient with mild native AS or mild-moderate native MS and no change in clinical status | 5 (22%) |
| 5. Patients who have isolated APC or PVC without other evidence of heart disease | 3 (13%) |
Overall, a large majority of TTEs (77% [ n = 199]) were found to be appropriate by the Echo AUC App ( Figure 1 ). The remainder of studies were Inappropriate (9% [ n = 23]) or Unclassified (14% [ n = 36], Table 4 ). Appropriate studies were more likely to be ordered by cardiologists compared with non-cardiologists (84%, 119/142 vs. 69%, 80/116; P = .01). Differences between cardiologist versus non-cardiologist ordering of Inappropriate (6%, 8/142 vs. 13%, 15/116; P = NS) and Unclassified studies (11%, 15/142 vs. 18%, 21/116; P = NS) did not meet statistical significance.
| Indication for TTE unclassified in the AUC for echocardiography | No. of studies (%) ( N = 36 unclassified studies) |
|---|---|
| Surveillance related to solid-organ transplant program | 11 (31%) |
| Heart Failure: revaluation of patient with heart failure (systolic or diastolic), >1 year since prior study with no change in clinical status | 6 (17%) |
| Preoperative evaluation | 6 (17%) |
| Native valvular stenosis or regurgitation: routine reevaluation with severity of valve disease or time interval for follow-up not specifically addressed by the AUC | 6 (17%) |
Appropriate studies were also more likely to be initial TTEs (90%, 152/168 of TTEs with no prior TTE were Appropriate vs. 52%, 47/90 of studies with prior TTEs; P < .001). Among studies with prior TTEs ( n = 90), those with prior TTEs within the last year were less frequently Appropriate compared with those for which the prior TTEs were more than 1 year prior (33%, 5/10 vs. 69%, 52/75; P = .04). A comparison of Inappropriate studies between those with prior studies within the last year and more than 1 year did not reach statistical significance (47%, 7/15 vs. 13%, 10/75; P = NS). Overall, 74% (17/23) of Inappropriate TTEs were studies for which prior TTEs had been performed. No significant difference in Unclassified designations occurred between studies with and without prior TTEs (18%, 16/90 vs. 12%, 20/168; P = NS).
Manually derived and Echo AUC App-derived appropriateness determinations were found to be similar (Appropriate: 77% by Echo AUC App vs. 81% by manual, kappa statistic 0.84 for Appropriate vs. Other; Inappropriate: 9% by Echo AUC App vs. 10% by manual, kappa statistic 0.95 for Inappropriate vs. Other; Unclassified: 14% by Echo AUC App vs. 9% by manual, kappa statistic 0.70 for Unclassified vs. Other). Overall, Echo AUC App-determined and manually determined classifications of indication/category (indications 1-51 or Unclassified) and overall levels of appropriateness (Appropriate, Inappropriate, or Unclassified) agreed in 82% ( n = 211, kappa statistic 0.79) and 94% ( n = 242, kappa statistic 0.83) of studies, respectively.
Overall, in 18% ( n = 47) of studies the manual rater and the Echo AUC App chose different individual indications (indications 1-51, or Unclassified). In 66% ( n = 31) of these, the ultimate appropriateness designation was not affected (A vs. I vs. Unclassified). Among these 31 studies, the most common indication pairing ( n = 12) was AUC indication 1 (“Symptoms suspected due to a cardiac etiology….”) and AUC indication 41 (Initial evaluation of known or suspected heart failure [systolic or diastolic]), both of which are Appropriate. In each of these 12 studies, the clinical scenario involved symptoms suggestive of heart failure; thus, both of these indications seemed to reasonably apply. Among the 16 studies for which the Echo AUC App-derived and manually derived appropriateness determinations were discrepant, the Echo AUC App found 2 manually determined Inappropriate studies to be Unclassified, 12 manually determined Appropriate studies to be Unclassified, and 2 manually determined Unclassified studies to be Appropriate. Thus, all studies for which there was a discrepant designation involved one rating methodology selecting an Unclassified designation. In no study did the two methods find a directly conflicting appropriateness designation (Appropriate vs. Inappropriate). The indications most commonly involved in a discrepant appropriateness designation were those requiring the rater to determine a “change in clinical status.” For these 16 studies with a discrepant designations, the Echo AUC App more frequently was the rating methodology to determine a study to be Unclassified (14/16 of discrepant studies).
The number of screens required to complete the Echo AUC App for all studies was 4.6 ± 1.3 (mean ± SD), with fewer screens required for TTEs deemed Appropriate (4.3 ± 1.3) and more screens for TTEs deemed Inappropriate (4.8 ± 1.2, P = .08 compared with Appropriate TTEs) and Unclassified (5.8 ± 1.1, P < .001 compared with Appropriate transesophageal echocardiograms; P < .01 compared with Inappropriate TTEs).
Echo AUC App completion time was collected automatically for all studies ( n = 258). Echo AUC App users achieved an appropriateness determination in a mean 55 ± SD 38 seconds (range 25-280 seconds). The median Echo AUC App completion time was 41 seconds, and 71% (184/258) of studies were completed in less than 1 minute. Echo AUC App completion times for TTEs deemed Appropriate ( n = 199), Inappropriate ( n = 23), and Unclassified ( n = 36) were 51 ± 31, 64 ± 28 ( P = .04 for comparison with Appropriate TTEs) and 74 ± 54 seconds ( P < .01 for comparison with Appropriate TTEs), respectively ( Table 5 ).
