This year marks a decade since leading cardiovascular organizations joined to develop the appropriate use criteria (AUC), which were aimed at improving clinical practice and outcomes for cardiovascular care. Born in part as a response to the recognition of rapid and possibly unsustainable increases in the number of cardiovascular procedures, especially those involving imaging, along with the observation of unexplained geographic variation in cardiovascular procedures, the AUC now address multiple cardiovascular services, including echocardiography. It is therefore now quite timely to ask, to what degree has the implementation of AUC in echocardiography improved clinical practice and outcomes? Unfortunately we likely lack a clear answer to this key question.

To begin connecting the use of the AUC for an imaging technology such as echocardiography to improvements in care, it is important to recognize that diagnostic decisions occur in the complex context of clinical evaluation and care, and that understanding the influence of the specific context of care delivery is necessary if we are ever to be able to determine ultimate clinical utility. The context of care is a key part of the global diagnostic system, which begins when patients experience symptoms and enter the care system, and concludes (at least initially) when they are given diagnoses to explain their symptoms and to direct care ( Figure 1 ). Ideally, these diagnoses are also explained to patients and referring physicians in clear terms. Sometimes diagnoses lead to changes in therapy, while in other cases, they reinforce prior thinking and reassure physicians (and perhaps patients) that the right decisions have been made. Reassurance is an elusive variable that may be difficult to measure, so a test that leads to no change in therapy may or may not be ultimately deemed useful, especially if a patient is at low risk for the possibility of serious disease.

This diagram of the diagnostic process illustrates the complexity of analyzing a subset of component parts without considering the larger system, including potential missing variables. Diagnostic testing, including echocardiography as an activity, is seen as a subcomponent of this greater process.

Reprinted with permission from the National Academy of Sciences, courtesy of the National Academies Press, Washington, D.C., 2015.

Reviewing Figure 1 , one can see that diagnostic testing occupies only one small part of the diagnostic and therapeutic continuum. A useful framework for analyzing the impact of the use of a diagnostic guide such as AUC on medical decision making and diagnostic efficacy was published by Fryback and Thornbury more than two decades ago. Their “six-tiered” model begins with a base level of measures of technical efficacy and incorporates steps of increasing complexity, including the efficacy of diagnostic accuracy, the efficacy of diagnostic thinking, therapeutic efficacy, patient outcome, and, finally, societal efficacy. If we narrow the focus to only one or two of these levels of efficacy and try to reach conclusions about the utility of the AUC by correlating appropriateness with such a limited view of efficacy, the results could be misleading, because of a failure to consider other correlations and confounding variables.

Operating within the confines of the diagnostic system, physicians and allied providers ideally use imaging studies judiciously to confirm or refute diagnostic hypotheses, in a process that is highly fluid and dynamic, sometimes leaving few durable artifacts beyond a brief differential diagnosis recorded in the medical record, or even less than that. The advent of the standardized electronic medical record has not necessarily led to an improved record of the diagnostic process, and in many cases it may in fact obscure the record of diagnostic reasoning. Once a diagnosis has been formulated, the physician and provider may be blinded to the number of diagnostic possibilities that preceded their decision because of hindsight bias, thus obscuring a key part of the process from analysis, unless it is prospectively sought and recorded before the conclusion of the testing process.

So we are left with examining measures of the diagnostic process that may or may not correlate with improved outcomes, in patient populations that are often less than perfectly characterized, and with a rather frail record of what actually occurred during this complex process. Although we can gain glimpses into how a tool such as the AUC may be useful in selecting patients for high-yield testing, we are like the occupants of Plato’s cave, who must reach conclusions more on the basis of the shadows on the reflective wall of our diagnostic testing system than a highly reliable picture of the full process as it is practiced every day in the United States. Our conclusions about the clinical utility of echocardiography on the basis of AUC measures are necessarily limited.

A provocative recent retrospective study suggested that among patients who met standard AUC for echocardiography, fewer than one third benefitted from changes in medical management due to the results of the test, while about one half had no changes made in management, suggesting that using AUC might overestimate overall clinical benefit. Reviewing the charts of 535 inpatients and outpatients selected from an initial group of 633 undergoing echocardiography at an academic medical center, Matulevicius et al . confirmed that more than 90% met criteria for appropriateness. Reviewers blinded to the AUC scores then made determinations of clinical efficacy. Effective echocardiographic studies were defined as an “active change in care” noted in relationship to the echocardiographic results, and lack of efficacy was characterized as “no change in care,” when there was no mention of the echocardiographic results in the record, prior knowledge of the patient’s condition, or another treatment process independent of the test results was already under way. The investigators also defined an intermediate category of “continuation of care,” in which there was no clear change in care, but the test results were documented in the chart. Although this retrospective study depended on an evaluation of medical records to determine utility, which within the complex system of patient diagnosis may well miss key steps in the diagnostic process, the finding that fewer than one third of patients appeared to have significant changes in management after undergoing echocardiographic examinations that met the AUC does raise important questions about the utility of AUC for stratifying patients for benefit of testing and the relationship of AUC to clinical efficacy. On the other hand, as Matulevicius et al . discussed subsequently in a letter to the editor, when the six-tiered model is considered, their study of the AUC focused on only two of the levels: the efficacy of diagnostic thinking and therapeutic efficacy. Without information from the other tiers, it is possible that this apparent weak link of AUC to changes in medical management might be due to a failure to consider how the use of AUC connects to other variables in other tiers, including patient outcomes and societal benefit.

In the current issue of JASE , Chiriac et al . report the results of a new study in which they sought to reevaluate the clinical benefit of echocardiographic testing in a well-characterized group of inpatients undergoing acute echocardiography within 3 days of admission, when one might expect these results to lead to significant changes in diagnosis and therapy. The investigators prospectively enrolled a subset of these patients on the basis of the ordering physicians’ completion of a simple online survey, in which they reported their individual perceptions of the utility of the test, noted if there were unexpected findings, and confirmed or not whether they would order the same echocardiographic study for other patients with similar findings. The investigators reviewed the patients’ medical records to seek confirmation of the ordering providers’ impressions, which reduced the chance of overestimating the clinical benefit solely on the basis of the providers’ survey responses. Once again, a very high percentage of patients, 95%, were found to meet criteria for appropriateness of use.

Among the 539 patients enrolled, Chiriac et al . found that nearly half of ordering physicians reported changes in management made in response to the results, a much stronger result than that found by Matulevicius et al . on the basis of retrospective record review, and noted a significant relationship between the degree of appropriateness of testing and the finding of a change in management. Additionally, one third of the tests detected findings deemed unexpected and clinically important by the ordering physicians, which were confirmed by record review. Almost all of the physicians completing surveys reported that the echocardiographic examinations had answered their clinical questions and that in similar circumstances, they would again order the test.

Is there an explanation for why such apparently contradictory conclusions were reached about the relationship between AUC and clinical efficacy in two similar groups of patients undergoing echocardiography? First, the severity of illness might have varied between the two studies, as Matulevicius et al . included outpatients, in whom the impact of testing changes might be expected to be lower, whereas Chiriac et al . studied inpatients during the first 3 days of admission, a time when changes in medical management are likely more common than for outpatients. Second, the specialties of ordering physicians differed between the two centers, implying that there may have been important differences in either the patient population or care context. Cardiologists ordered fewer than one third of echocardiograms in the study of Matulevicius et al ., whereas they constituted one half of the ordering physicians in Chiriac et al .’s study, in which a higher percentage of tests were deemed efficacious. It is possible that cardiologists are more adept at ordering “appropriate” echocardiographic studies than noncardiologists, because of both their knowledge of the criteria and perhaps their ability to select patients with the highest yield for other reasons. Additionally, the two groups used different versions of the AUC, and there is reason to believe that the more recent updated AUC may be more definitive. A limitation common to both studies was the inclusion of only a modest number of patients with “inappropriate” indications, which could have led to limitations in statistical power when seeking to compare and contrast the “appropriate” and “inappropriate” groups.

The most contentious area for discussion is how we are to interpret the significance of the substantial group Matulevicius et al . labeled “continuation of current care.” When a patient admitted with seemingly obvious findings of congestive heart failure and cardiomegaly on initial evaluation later undergoes echocardiography that confirms the presence of systolic dysfunction, although there may be no change in management, few would argue that this was not an effective test (as long as there was no other recent confirmatory testing performed). It is also important to acknowledge that the absence of findings of cardiac abnormalities in patients with conditions such as unexplained dyspnea can also be clinically useful, by highlighting the need to search elsewhere for an alternative explanation and treatment. However, excessive use of echocardiography to confirm or refute the presence of abnormal cardiac findings in patients with lower risk disease states or uncommon conditions could easily lead to overuse of echocardiography. This highlights an important distinction: not all appropriately ordered echocardiographic examinations are definitively necessary, and clinical necessity is therefore an incompletely unstudied variable in both of these interesting studies, depending in part on the acuity and severity of illness of the individual patients.

What we can safely conclude from the interesting results of Matulevicius et al . and Chiriac et al . is that we have much more work to do be done before we reach broad conclusions about the utility of AUC. Perhaps a yield of changes in management following appropriately ordered echocardiography of 30% to 50% is not at all unreasonable, even if we withhold judgment on the subset of patients in whom management was continued unchanged. If echocardiography is ordered and there is no chart documentation that its results were considered by the ordering physician or discussed with patients, then we should question whether such testing should be performed, whether AUC are met or not.

While acknowledging the need for randomized clinical trials of the utility of echocardiography to reach more substantive conclusions, such multicenter studies will be expensive and difficult to conduct. In the interim, we can specify how smaller trials might look. First, prospective studies should ideally be performed on well-characterized patient populations eligible for echocardiography, perhaps restricting the studies to a small number of the AUC, those accounting for a high proportion of echocardiographic examinations, rather than lumping together so many disparate reasons for echocardiography contained within the full extent of AUC. In addition to specifying the reason for testing, ordering physicians could be asked to list the pretest likelihood of the anticipated findings before they are informed of the results. After the release of results, they would be asked to identify the incremental value of the test result on the basis of the current clinical information about the patient. Incremental value might be defined as change in therapy, recognition of need for additional testing, or unexpected findings; provider responses should be confirmed by record review. Should the incremental value be defined as confirming current management or patient reassurance, the percentage of studies thus classified could be compared among several study centers to identify significant geographic variations.

It is logical to conclude that sometimes, confirmation of current therapy and patient reassurance are acceptable results of diagnostic testing, as Chiriac et al . have noted, as long as the modality is not overused in patients with low-risk conditions, for whom reassurance is unnecessary. Overuse in these patients may be detected by identifying provider groups that order disproportionately large numbers of studies with low pretest likelihood estimates or those who commonly prespecify a high pretest likelihood of detecting an abnormality that is rarely found, particularly for lower risk conditions. Ideally, individual ordering physicians can improve our own efficacy of ordering by tracking the relationship between our prespecified likelihood of finding an expected abnormality, and comparing this “clinical batting average” to the results of testing on an ongoing basis.

While we learn how better to define the clinical utility of echocardiography, gather more data, and continue to find ways to increase testing efficacy, we should not allow minor differences in current studies of the AUC to diminish the importance of these criteria, which will continue to evolve and improve. Whatever the downstream impact on clinical efficacy in the complex diagnostic system may be, the use of AUC by ordering providers ensures at a minimum that one is at least pausing (however briefly) to reconsider the need for testing, on the basis of expert guidelines. All of this might simply be grist for the mill of academic discussion were it not for the elephant in the room: the anxious payer who in seeking methods to curb the rising costs of noninvasive diagnostic imaging might incorrectly grasp at the convenience of the AUC tool to achieve this aim by restricting payment for echocardiographic examinations that fail to demonstrate clear changes in the management of their insured, or refusing payment for the rare examination that is clinically effective while not meeting the standard AUC. On the basis of available data, there is no reliable evidence to justify such decisions.