Medical practice has now enteredthe era of “evidence-based medicine,” characterized by an increasing societal belief that recommendations about the use of treatments in clinical practice should be based on scientific information rather than intuition or opinion. As our society’s tolerance for increasing the proportion of financial resources spent on medical care has been exhausted, the only rational way to allocate resources is to understand whether competing therapeutic approaches provide clinical benefit and, if so, the cost required to achieve that benefit. Simultaneous with the realization that expansion of medical financial resources is not limitless, the huge societal investment in biotechnology is beginning to pay off in the form of many potential new approaches to treating disease. Therefore, with current methodology, the need for evidence is increasing faster than the resources are being made available to perform the studies.

This accelerating pace of quantitative research creates a need for practicing physicians to understand the methods of evaluating clinical evidence well beyond the level taught in most medical schools. The plethora of new studies and the increasing questioning of medical decision-making point to the benefits of a better understanding by practitioners of the critical issues in clinical research design. Because of the high visibility and quantitative nature of cardiology, the importance of developing these skills among cardiologists is paramount.

The first chapter of the book of Daniel in the Bible describes a comparative experiment in which Daniel urges a servant to experiment with refraining from eating the king’s food, and, indeed, the healthier diet is associated with the finding that “at the end of ten days their countenances appeared fairer and they were fatter in flesh, than all the youths that did eat of the king’s food.” The modern human clinical trial, however, was not pioneered until 1747, when James Lind tried six different treatments for scurvy in 12 sailors; despite the small sample size, the virtues of oranges and lemons were evident (1). However, partly because of inadequate sample size, he failed to recognize the primary beneficial effect of oranges and lemons, considering “pure dry air” to be the most beneficial intervention.

In the nineteenth century P. C. A. Louis, in a treatise entitled La Methode Numerique, developed the concept of comparative observational studies (2,3). In casting doubt on the common practice of bloodletting, he clearly identified the concept of confounding, and he emphasized the need to have comparable groups when making comparisons. In the 1840s, Semmelweis (4) noted that maternal mortality was higher in women giving birth in hospitals in which deliveries were attended by physicians than in women giving birth in hospitals in which deliveries were attended by midwives. He succeeded in demonstrating that the excess mortality was related to contamination from autopsies and succeeded in separating autopsy teaching from attending childbirth by physicians, which led to a drop in maternal mortality.

The first randomization was performed by Fisher and Mackenzie in 1926 in an agricultural study (5). In developing analysis of variance, they recognized that experimental observations must be independent and not confounded to allow full acceptance of the statistical methodology. They therefore randomized different plots to different approaches to the application of fertilizer. Amberson is credited with the first randomization of patients, in a 1931 trial of tuberculosis therapy in 24 patients, using a coin toss to make treatment assignments (1). The British Medical Research Council trial of streptomycin in the treatment of tuberculosis began the modern era of clinical trials in 1948 (6). This trial established the principles of the use of randomization in large numbers of patients and set guidelines for administration of the experimental therapy and objective evaluation of outcomes.

Modern outcomes research was presaged by important observations from Florence Nightingale, who observed a higher mortality in hospitals than out of hospitals in the treatment of amputation. Her notes provide a detailed accounting of observational study methodology in the adjustment for differences in outcome as a function of baseline characteristics.

In 1914, Codman (7) wrote a treatise about measurement of outcomes that provides a classic discussion of the need for hospitals to evaluate outcomes. He proposed that reimbursement should be based on demonstration of results. Unfortunately, his message was not popular, and his hospital went bankrupt.

In the last decade, computers have enabled rapid accumulation of data in thousands of patients in studies conducted throughout the world. R. Peto, S. Yusuf, P. Sleight, and R. Collins developed the concept of the large, simple trial in ISIS-I (8), beginning with the concept that only by randomizing 10,000 patients could the beneficial effects of β-blockers be understood. The development of client server architecture provided a mechanism for aggregating large amounts of data and distributing them quickly to multiple users. Finally, the most recent advances in the development of the World Wide Web provide an opportunity to share information instantaneously in multiple locations around the world.

Evaluating therapies and interpreting the results as they are presented requires an understanding of the goals of the investigation; these goals can be conveniently categorized in a similar manner to the nomenclature used by the U.S. Food and Drug Administration to characterize the phase of investigation in clinical trials (Table 43.1). Conclusions drawn from a small pilot study of a therapy never before given to humans should be quite different from conclusions drawn from a study measuring clinical outcomes comparing two therapies used in current practice; thus regulatory authorities have developed a nomenclature for discussing these phases of investigation.

When a new therapy is devised, it must first be studied in normal, healthy volunteers to ensure that no unexpected side effects will occur. This type of study is commonly referred to as Phase I. In some cases, when concern exists that the therapy may be highly toxic, patients at the advanced stage of disease (with presumably little to lose and much to gain if the treatment is successful) are evaluated instead. Many novel therapies affecting the coagulation system in which the risk of bleeding would not be acceptable for a normal volunteer fall into this category. The most extreme example of this approach was the use of recently deceased subjects (“neomorts”) in the initial studies of glycoprotein IIb/IIIa inhibitors (9). Although Phase I trials may generate substantial enthusiasm, they should not be used to make recommendations for routine clinical practice.

After the new therapy demonstrates some physiologic effect in the first phase of testing, the therapy must, in Phase II, demonstrate both desired physiologic activity and evidence of clinical benefit in the clinically relevant disease, at least from the perspective of “surrogate” pathophysiologic endpoints. For example, a heart failure therapy would need to demonstrate improvement in cardiac output, left ventricular function, or neurohormonal status. Often investigators are tempted to use these surrogate endpoints to recommend major changes in practice, but multiple experiences have demonstrated the danger of this approach to recommending changes in clinical practice (10). Often, even if the surrogate endpoints appear to be improved, large enough trials may show that the clinical outcomes are adversely affected. One of the major unresolved conceptual issues in modern cardiovascular research is how to use Phase II data to make wise choices about proceeding to the next phase.

The third phase, commonly referred to as the “pivotal” phase, evaluates the therapy in the relevant clinical context with the goal of determining whether the therapy should be used in clinical practice. For Phase III, the relevant endpoints include measures that can be recognized by patients as important: survival time, major clinical events, quality of life, and cost. A well-designed clinical trial with a positive effect on clinical outcomes justifies serious consideration for a change in clinical practice and certainly provides grounds for regulatory approval for sales and marketing.

After a therapy is in use and approved by regulatory authorities, Phase IV begins. Traditionally, Phase IV has been viewed as the monitoring of the use of a therapy in clinical practice, with a responsibility of developing more effective protocols for the use of that therapy, based on inference from observations and reporting of adverse events. The importance of this phase has evolved from the recognition that many circumstances
experienced in clinical practice will not have been encountered in randomized trials completed at the time of regulatory approval. Examples of Phase IV studies include the evaluation of new dosing regimens, as in several ongoing comparisons of low-dose versus high-dose angiotensin-converting-enzyme inhibition in patients with heart failure and the prospective registries of use of therapies such as the National Registry of Myocardial Infarction (NRMI) (11). Recently, with an increasing array of effective therapies, Phase IV is viewed as a time to compare one effective marketed therapy against another. In some cases this need arises because of changing doses or expanding indications for a therapy, whereas in other cases, the Phase III trials did not provide the relevant comparisons for a particular therapeutic comparison. GUSTO-I is an example of a Phase III trial in which a new dosing regimen of alteplase (accelerated dosing) was compared with a standard dose of streptokinase (12). The ongoing GUSTO-III Trial is an example of a Phase IV trial initiated because reteplase was approved by regulatory authorities based on a comparison with streptokinase, but it was not compared with accelerated alteplase; thus, GUSTO-III is performing that comparison because accelerated alteplase is the reference standard for practice in the United States.

With rare exceptions, the purpose of a Phase III or Phase IV clinical trial or outcome study is to estimate what is likely to happen to the next patient given one treatment or the other. To assess the degree to which the proposed study enhances the ability to understand what will happen to the next patient, the investigator must be aware of an array of methodologic and clinical issues. Although this task requires substantial expertise and experience, the issues can be considered in a broad framework. The broadest, but most essential concepts in understanding the relevance of a clinical study to practice are the concepts of validity and generalizability. Table 43.2 illustrates an approach to these issues, developed by the McMaster group, to be used when reading the literature.

The manner in which the results of clinical research are reported can profoundly influence the perception of practitioners using the information to decide which therapies to use. A clinical trial will produce a different degree of enthusiasm about the therapy tested when the results are presented in the most favorable light, even when the exact results are provided in addition to the risk reduction so that the practitioner could reconstruct the results in different ways (13). Multiple studies (14,15) have demonstrated that physicians are much more likely to recommend prescribing a therapy when the results are presented as a relative risk reduction rather than an absolute difference in outcomes.

A recent example of a trial that has evoked considerable interest based on a predominant depiction of relative treatment effect is the 4S trial comparing simvastatin with placebo in patients with elevated low-density-lipoprotein (LDL) cholesterol and a history of ischemic heart disease (16). The results have been mostly depicted as a 30% reduction in overall mortality and a 42% reduction in coronary mortality. Rarely are the results presented in terms of absolute differences (approximately 0.6% reduction in mortality per year). This difference is highly significant because more than 4,000 patients were randomized; the cost-effectiveness data support use of simvastatin in this situation (18). In contrast, the Bypass Angioplasty Revascularization Investigation (BARI) found almost exactly the same difference between coronary artery bypass grafting and percutaneous revascularization, but with just more than 1,800 patients randomized, this difference was not statistically significant, and the conclusion was reached that no major differences in outcome exist between the two technologies in patients with suitable coronary anatomy for either. If the same result had been achieved with 4,000 patients randomized, would the same conclusion have been drawn as for simvastatin?

The results of pragmatic clinical trials are most appropriately expressed in terms of the number of poor outcomes prevented by the more effective treatment per 100 or 1,000 patients treated. This measure, representing the absolute benefit of therapy, number needed to treat (NNT), translates the results for specific populations studied into public health terms by quantifying how many patients would need to be treated to create a specific health benefit. The absolute difference can be used to assess quantitative interactions, that is, significant differences in the number of patients needed to treat to achieve a degree of benefit with a therapy as a function of the type of patient treated. An example is the use of thrombolytic therapy; the Fibrinolytic Therapy Trialists’ (FTT) Collaborative Group demonstrated that 37 lives per 1,000 patients treated are saved when thrombolytic therapy is used in patients with anterior ST-segment elevation, whereas only 8 lives per 1,000 are saved when patients with inferior ST-segment elevation are treated (Fig. 43.1) (19). The direction of the treatment effect is the same, but the magnitude of the effect is different.

This approach becomes more complex with endpoints that are not discrete, such as exercise time. One approach to expressing trial results when the endpoint is a continuous measurement is to define the minimal clinically important difference (the smallest difference that would lead practitioners to change their practices) and to express the results in terms of the NNT to achieve that minimal clinically important difference.

The relative benefit of therapy, on the other hand, is the best measure of the treatment effect in biologic terms. This concept is defined as the proportional reduction in risk resulting from the more effective treatment and is generally expressed in terms of an odds ratio or relative risk reduction. The relative treatment effect can be used to assess qualitative interactions, which represent statistically significant differences in the direction of the treatment effect as a function of the type of patient treated. In the FTT analysis (19), the treatment effect in patients without ST-segment elevation is heterogeneous compared with patients with ST-segment elevation. Figure 43.2 gives the calculations for commonly used measures of treatment effect. A particularly useful display of data is the odds ratio plot, as shown in Figure 43.3. Both absolute and relative differences in outcome should be expressed in terms of point estimates and confidence intervals. This type of display gives the reader a balanced perspective because both the relative and absolute differences are important as well as the confidence in the estimate. Without the confidence intervals, the reader has difficulty knowing the precision of the estimate of the treatment effect.

As experience with multiple clinical trials is accumulating, some general concepts seem worth emphasizing. These generalities do not always pertain, but they serve as useful guides to the design or interpretation of trials.

When reading the results of a clinical study or designing a study, the purpose of the investigation is critical to placing the outcome in context. Those who design the investigation have the responsibility of constructing the project and presenting the results in a manner reflecting the intent of the study. In a small Phase II study, an improvement in a surrogate pathophysiologic outcome is exciting and could easily lead the investigator to overstate the clinical implications of the finding. Similarly, megatrials with little data collection seldom give useful information about disease mechanisms unless carefully planned substudies are performed. The structural characteristics of trials can be characterized as a function of the following attributes.