In health care, information is a collection of material, workflow documentation, and recorded observations (see Section II ). This information is largely in electronic (computer) format.

Data consist of organized values for variables, usually expressed symbolically (e.g., numerically) by means of a controlled vocabulary (see Section III ). Characterization of data includes descriptive statistics that summarize the data and express their variability.

Analysis is a process, using a large repertoire of methods, by which data are explored, important findings are revealed and unimportant ones suppressed, and relations are clarified and quantified (see Section IV ).

Knowledge is the synthesis of information, data, and analyses arrived at by application of information, data, and analyses in the form of study designs that yield evidence for evidence-based medicine based on average treatment effects and evidence for precision medicine based on individual treatment effects (see Section V ). New knowledge may take the form of clinical inferences , which are summarizing statements that synthesize information, data, and analyses, drawn with varying degrees of confidence that they are true. It may also include speculations , which are statements suggested by the data or by reasoning, often about mechanisms, without direct supportive data. Ideally, it also includes new hypotheses , which are testable statements suggested by reasoning, inferences, or speculations from the information, data, and analyses.

To be useful both to incrementally benefit science and to contribute to advancement of health and health care, knowledge must be disseminated. The clearest example of this is scientific publications and presentations (see Section VI ). That is, to be most useful, what has been found must be made public. In this edition, we only hint at the rapidly evolving field of implementation and dissemination science and its barriers (see Section VII ), but this science is crucial to apply knowledge to routine clinical practice. The lag between discovery and its integration into routine clinical practice is estimated to be 17 years, and this needs to be shortened.

Slips are failures in execution of actions and are commonly associated with attention failures ( Box 7.2 ). Some external stimulus interrupts a sequence of actions or in some other way intrudes into them such that attention is redirected. In that instance, the intended action is not taken. Lapses are failures of memory. A step in the plan is omitted, one’s place in a sequence of actions is lost, or the reason for what one is doing is forgotten. Mistakes relate to plans and so take two familiar forms: (1) misapplication to the immediate situation of a good plan (rule) appropriate for a different and more usual situation and (2) application of a wrong plan (rule).

Slips and lapses constitute active errors . They occur at the physician–patient interface. Mistakes, in addition, constitute many latent errors . These are indirect errors that relate to performance by leaders, decision makers, managers, certifying boards, environmental services, and a host of activities that share a common trait: planning, decisions, ideas, and philosophy removed in time and space from the immediate healthcare environment in which the error occurred. These are a category of errors over which the surgeon caring for a patient has little or no control or chance of modifying because latent errors are embedded in the system. It is claimed by students of human error in other contexts that the greatest chance of preventing adverse outcomes from human error is in discovering and neutralizing latent error.

If one considers all the possibilities for error in daily life, what is remarkable is that so few are made. We are surrounded with unimaginable complexity, yet we cope nearly perfectly because our minds simplify complex information. Think of how remarkably accident-free our early-morning commutes to the hospital are while driving complex machines in complex traffic patterns while we plan our day and listen to the news, weather report, and commercials.

When this cognitive strategy fails, it does so in only a few stereotypical ways. Because of this, models have been developed, based largely on observation of human error, that mimic human behavior by incorporating a fallible information-handling device (our minds) that operates correctly nearly always, but is occasionally wrong. Central to the theory on which these models are based is that our minds can remarkably simplify complex information. Exceedingly rare imperfect performance is theorized to be the price we pay for being able to cope this way with complexity. The mechanisms of human error are purported to stem from three aspects of “fallible machines”: downregulation, upregulation, and primitive mechanisms of information retrieval. In the text that follows, we borrow heavily from the human factors work of James Reason.

We have presented this cognitive model in part because it suggests constructive steps for reducing human error and, thus, surgical failure.

It affirms the necessity for intense apprentice-type training that leads to automatization of surgical skills and problem-solving rules. It equally suggests the value of simulators for acquiring such skills. It supports creating an environment that minimizes or masks potential distractions. It supports a system that discovers errors and allows recovery from them before injury occurs. This requires a well-trained team in which each individual is familiar with the operative protocol and is alert to any departures from it. In this regard, de Leval and colleagues’ findings are sobering. ^, Major errors were often recognized and corrected by the surgical team, but minor ones were not, and a higher number of minor errors was strongly associated with adverse outcomes. It is also sobering that self-reporting of intraoperative errors was of no value. Must there be a human factors professional at the elbow of every surgeon?

James Reason suggested that “cognitive prostheses” may be of value, some of which are being advocated in medicine. For example, there is much that computers can do to reduce medication errors. A prime target is knowledge-based errors. Reducing these errors may be achievable through computer-based artificial intelligence, and novel modes of information assembly, processing, and display for processing by the human mind. Finally, if latent errors are the root cause of many active errors, analysis and correction at the system level will be required. A cardiac surgery program may fail, for example, from latent errors traceable to management of the blood bank, postoperative care practices, ventilation systems, and even complex administrative decisions at the level of hospitals, universities with which they may be associated, and national health system policies and regulations within which they operate.

Every aspect of cardiac surgery has been described in thousands of publications about technical details. These may be techniques of preoperative evaluation, such as imaging, preoperative and intraoperative aspects of anesthesia, technical details of operations, details of postoperative management, and patient recovery. Many describe new devices and their properties and even failures, and some even describe novel methods of analysis.

The common denominator of these publications is transparent disclosure of techniques for their use by all in the field. Advancements by one surgeon or group that are not shared is a loss to the entire field of cardiac surgery.

The impact factor of most journals is not enhanced by individual case reports or very short case series because they are not commonly cited. However, particularly in congenital heart disease, these reports often advance the field. They may initially appear as oddities, but may later fit into gaps of knowledge, or their successful repair may advance the field in unexpected ways.

Whether a short case series, a thousand-case series, or a national case series or hundreds of thousands of more cases, what are often classified as “just” descriptive studies have advanced the field of cardiac surgery tremendously. Hemodynamics of prosthetic valves, experience with the radial artery as a conduit for coronary bypass grafting, national experience with ablation for atrial fibrillation, Medicare costs, heart failure rehospitalizations after certain operations, and quality of life after aneurysm repair are all examples of descriptive case series. In “Technique for Successful Clinical Research” that follows, we will describe how to design effective descriptive studies that answer important relevant questions that are hypothesis driven and hypothesis generating.

In contrast to studies that describe a series of cases are observational studies that compare outcomes of different treatments. The fundamental objection to using observational clinical data for comparing treatments is that many uncontrolled variables affect outcome. Thus, attributing outcome differences to just one factor—the alternative treatment—stretches credibility. Even a cursory glance at the characteristics of patients treated one way versus another usually reveals that they are different groups. This should be expected because treatment has been selected by experts who believe they know what is best for a given patient. The accusation that one is comparing apples and oranges is well justified!

Indeed, a consistent message since Graunt is that risk factors for outcomes from analyses of clinical experience (and these include treatment differences) are associations , not causal relations . Multivariable adjustment for differences in outcome is valuable, and methods for multivariable analysis will be detailed in Section IV under “Multivariable Analysis.” It must be pointed out, however, that if alternative therapies are being analyzed by multivariable analysis, these analyses are not guaranteed to be effective in eliminating selection bias as the genesis of a difference in outcome (a form of confounding ). ^{, ,}

Over the years, a number of attempts have been made to move “association” toward “causality.” One such method is the case-control study. ^, The method seems logical and straightforward in concept. Patients in one treatment group (cases) are matched with one or more patients in the other treatment group (controls) according to variables such as age, sex, and ventricular function. However, case matching is rarely easy in practice. How closely matched must the pair of patients be in age? How close in ejection fraction? “We don’t have anyone to match this patient in both age and ejection fraction!” The more variables that must be matched, the more difficult it is to find a match in all specified characteristics. Yet matching on only a few variables may not protect well against apples-and-oranges comparisons. Diabolically, selection factor effects, which case matching is intended to reduce, may increase bias when unmatched cases are simply eliminated.

During the 1980s, federal support for complex clinical trials in heart disease was abundant. Perhaps as a result, few of us noticed the important advances being made in statistical methods for valid, nonrandomized comparisons. One example was the seminal 1983 Biometrika paper “The Central Role of the Propensity Score in Observational Studies for Causal Effects” by Rosenbaum and Rubin. In the 1990s, as the funding climate changed, interest in methods for making nonrandomized comparisons accelerated. This interest has increased as comparative effectiveness research has taken on greater importance, and the concept of a Learning Health System has been advocated by the National Academy of Medicine in the United States. Studies using propensity scores for comparison of outcomes are detailed in Section V under “Comparisons Based on the Propensity Score.”

More recently, formal statistical methods for conceptually treating patients as their own controls—something that has been done in cardiac surgery since the 1970s ^{, ,} —is known as virtual twin methodology. ^, The method borrows the concept of propensity score methods in identifying patients empirically eligible to be treated by two or more therapies, but both refines and importantly extends those methods such that patients are matched to themselves first with treatment received and then with the counterfactual treatment. Clinical research using virtual twins is detailed in Section V under “Virtual Twins and Causal Analysis.”

A proposal first serves to clarify and bring into focus the question or set of related questions being asked. A common mistake is to ask questions that are unfocused, or uninteresting, or overworked, or that do not target a clear gap in knowledge or area of importance. Marbán and Braunwald say “Ask a bold question…about which you can feel passionate.”

The next step is to clearly define the inclusion and exclusion criteria for the study group . A common mistake is to define this group too narrowly, such that cases “fall through the cracks” or an insufficient spectrum is stipulated. The inclusive dates should be considered carefully. Readers will be suspicious if the dates are “strange”; did you stop just before a series of deaths? Whole years or half years dispel these suspicions. Similarly, suspicion arises when a study consists of a “nice” number of patients, such as “the first 100 or 1000 repairs.”

In defining the study group, particular care should be taken to include the denominator. For example, a study question may relate to postoperative neurologic events, but it is also important to have a denominator to put these events into context. Or one may wish to evaluate a new surgical technique but will be unable to compare it with the standard technique without a comparison group. A study of only numerators is the true definition of a retrospective study; if the denominator is included, it is a prospective or cohort study ( Box 7.5 ). ^, The inclusion and exclusion criteria should allow one to perform an initial search for patients as a feasibility study. Are there too few patients to answer the study question? A common failing is forgetting that if an outcome event is the endpoint, the effective sample size is the number of events observed (see Box 7.4 ). A study may have 1000 patients, but if only 10 events are observed, one cannot find multiple risk factors for those events. In cardiothoracic surgery, the number of events is often a very small fraction of the denominator, and such data are known as imbalanced. This is particularly important for nonparametric machine learning methods and in applying traditional goodness-of-fit statistics (like the C-statistic) to parametric models.

The deliverable for this feasibility investigation is a CONSORT-style diagram ( http://www.consort-statement.org/consort-statement/flow-diagram ). For observational studies, there is a related STROBE guideline, and all these are part of the EQUATOR (Enhancing the QUA lity and T ransparency O f health R esearch) network ( www.equator-network.org ).

Endpoints (results, outcomes) should be linked one-to-one to the study questions and accompanying hypotheses. For example, if the first subquestion of a comparative effectiveness study is “How do these treatment groups differ?” and the mechanistic hypothesis is that for some patients, experienced, knowledgeable clinicians have selected one treatment for them and selected another treatment for other patients. However, for a group of these patients, systematic selection has not occurred, creating among them virtual equipoise. The endpoint for that hypothesis is the treatment received. For hypotheses about effectiveness of the treatment, endpoints would be those about short-, medium-, or long-term effectiveness of treatment. For a hypothesis related to differences in safety of the therapies, perioperative complications are likely among the endpoints. Whatever the endpoint, it must be clearly defined in a reproducible fashion. Generally, every event endpoint should be accompanied by its date of occurrence. A common failing is that for repeated endpoints (e.g., thromboembolism, assessments of functional status, rehospitalizations, echocardiographic assessment of valve gradient or grade of regurgitation), only the first or most recent time they occurred or were assessed is recorded. The latter is particularly egregious because every patient has a different interval from, say, surgery to that last observation, so the data at last observation are not interpretable. Instead, record every instance, every assessment. Techniques to analyze repeated endpoints are available (see “ Longitudinal Outcomes ” in Section IV).

Careful attention must be paid to the covariables that will be studied. They should be pertinent to the study question (purpose, objective, hypothesis). A common failing is to collect values for too many variables such that quality of data collection for important variables suffers. This error usually arises in a reasonable and understandable way. The surgeon-investigator reasons that because the patient medical records must be reviewed, a number of other variables “may as well” be abstracted while there. Or realizing the full complexity of the clinical setting, the surgeon-investigator feels compelled to collect information on all possible ramifications of the study, even if some of it is peripheral to the study’s focus. John Kirklin called this “the Christmas tree effect,” meaning adding ornament upon ornament until they dominate what once was “just” a fine tree. There needs to be a balance between so sparse a set of variables that little can be done by way of risk factor identification or balancing characteristics of the group, and so rich a set of variables that the study flounders or insufficient care is given to the quality and completeness of relevant variables.

A data analysis plan should be linked one-to-one with the study questions, their accompanying hypotheses, and their accompanying endpoints. For example, if the first subquestion relates to how patients are receiving one therapy versus another, the first objective in the analysis plan would be comparing characteristics of the two groups of patients, for example, by a table or figure of standardized differences. The second step would be to develop a parsimonious model of these differences that would then be augmented into a nonparsimonious model, followed by generating a balancing or propensity score. This would be followed by 1:1 or weighted matching and comparison of characteristics of the resulting matched groups. This, in turn, would lead to comparisons of endpoints, and finally study of the remaining unmatched cases (the “oranges”).

Every study has limitations and anticipated problems. These can be identified by a brief but serious investigation of the state of all the above. If any appear insurmountable or present fatal flaws that preclude later publication, the study should be abandoned. There are always more questions than can be addressed in cardiac surgery, so not being able to answer some specific research question is not an excuse to abandon the search for new knowledge!

Every aspect of the data analysis plan is likely to generate perhaps some simple answers, but more often tables and figures. Laying these out with the proposal will be helpful to the statisticians analyzing the data and often identifies items missing from the analysis plan—those analyses, tables, and figures that “connect the dots.”

Develop a timetable for data abstraction, data set generation, data analysis and reporting, possible meeting abstract and deadline, and all deliverables at various mileposts in the study. If the timetable is beyond that tolerable, abandon the study. It is rare for a study to be completed in a year from start to finish. This emphasizes both the bottlenecks of research and the need for lifelong commitment. Although abstract deadlines often drive the timetable, this is a poor milepost (see “ Presentation ” in Section V).

The final deliverable is a completed research proposal (or protocol) that is ready for review by collaborating investigators, data managers, and statisticians, followed by revision and iterative refinement of the proposal. The better the proposal, the better the research and the easier for statisticians to analyze. The protocol will often need to be approved by a research committee for its scientific merit and funding, and by the institutional review (ethics) board.

The proposal should be a living document. It is likely to be updated throughout the course of a study, and we advocate online tracking of each study, with periodic updates of the protocol as one of the tasks in project management.

In medicine, information is multidimensional. A given value for a variable may carry with it time, who or what machine generated the value, the context of obtaining the value (“documentation”), format or units of measurement, and a host of attributes and relationships—indeed ontology—that give the value meaning within the context of healthcare delivery. Simply storing a set of values is insufficient. At a minimum, a comprehensive data dictionary must be developed, and ideally the database structure will contain metadata as to who entered the data, completeness of the data, and who can access and use the data. When data comes to analysis, information relevant to the values may importantly affect the analysis.

In relational database technology, variables are arranged as columns of a table, sets of columns are organized as a table, individual patients are in rows, and a set of interrelated tables constitute the database ( Fig. 7.3 A). Retrieval of data from such a model is often accompanied by the structured query language (SQL).

Comparison of relational information model with a semistructured one presented as a directed acyclic graph. (A) Relational. Tables are related by ID and source. Note that second table is many-to-one; that is, many postoperative echocardiograms were performed on one patient. (B) Semistructured.

(From Jonathan Borden, www.jonathanborden-md.com .)

Popularity of the relational model among clinical researchers stems from its simplicity in handling a microscopic corner of medical information. As soon as a new topic is addressed or new variables must be collected, the typical behavior of the research team is to generate a new specific database. Rarely do these multiple, independent, and to some extent redundant databases communicate with one another across studies. Thus, simplicity can work against more complex or comprehensive future studies.

A different kind of information model emerged from an important conference at UAB of leaders in the development of several different types of database as part of a CPR project. After review of the strengths and limitations of various information models, a novel approach was suggested by Kirklin and then formalized. He proposed that all information that provided context and meaning to a value for a variable be packaged together. He envisioned that such a complex data element should be able to reside as an independent self-sufficient entity.

This idea has several meritorious implications. First, an electronic container for a collection of complex data elements could consist of a highly stable, totally generic repository for a CPR because it would be required to possess no knowledge of content of any data element. It could therefore manage important information storage and retrieval functions, implement data encryption for privacy and confidentiality, store knowledge bases used to construct the complex data elements and retrieve them, maintain audit trails, and perform all those functions of database management systems that are independent of data content. The second implication is that as medical knowledge increases, new entries would be made in the knowledge-based dictionaries. These would be updated, not the database structure. Not only would this ease database maintenance, but it would also enforce documentation in the knowledge base. The third implication, and the one most important for clinical research, is that no a priori limitations would be placed on relations; they could be of any dimensionality considered useful at the time data elements were retrieved for analysis. Thus, the electronic container is a single variable value-pair augmented with contextual documentation and capable of being modified as new or more knowledge accrues ( Fig. 7.3 B).

Essential characteristics of such an information repository would be:

• •

  Self-documentation at the level of individual values for a variable (complex data element)

• •

  Self-reporting at the time of data element retrieval and potential

• •

  Self-displaying in a human-computer interface

• •

  Self-organizing

The latter is an important attribute for future implementation of what might be called “artificial intelligence” features of a CPR. These may be as simple as self-generation of alerts, solution of multivariable equations or algorithms for decision support at the individual patient level, or intelligent data mining for undiscovered relations within the information.

About 1995, at the time these ideas were being developed at UAB, similar thinking was going on among computer scientists at Stanford University and the University of Pennsylvania, arising from different stimuli. ^, They termed an information model of complex data elements that carried with them all attributes intended for self-documentation, self-reporting, and self-organizing semistructured data. This phrase meant that the data elements were fully structured, but no necessary relation of one data element to another was presupposed. The culmination of these efforts was a database for storing complex data elements called Lore and a novel query language for retrieving complex data elements called Lorel.

In the 1990s, it was recognized that the information structure suggested by Kirklin and the University of Pennsylvania and Stanford computer scientists could be conceptualized as a directed acyclic graph. At that time, another entity was also rapidly coming into existence with similar properties, but of global proportions: the World Wide Web (WWW, or simply the Web). A Web page is analogous to a complex data element, with an essential feature being that it is self-describing, so it can be retrieved. The Web is the infrastructure for these pages. It has no need to be aware of Web page content. The subject matter has no bounds. Not surprisingly, then, the tools developed for retrieving semistructured data were quickly adapted to what has become known as search engines for the Web. Like Dr. Kirklin’s vision of complex data elements, information retrieved by a search engine can become related in ways never envisioned by the person generating it, because full structure is imposed only at the time of retrieval, not at the time of storage.

Thus, at Cleveland Clinic, investigators both harnessed and developed Semantic Web tools for data storage and manipulation, in part within the framework of the World Wide Web Consortium (W3C) and in part through ontologies built by Douglas Lenat of Cycorp.

An information model was built on a graph-based data model known as the Resource Description Framework (RDF), as well as a framework for describing conceptual models of RDF data in a particular domain known as the Ontology Web Language (OWL). It also includes a standard querying language called SPARQL .

RDF captures meaning as a collection of triples consisting of components analogous to those of an elementary sentence in natural language: subject, verb, object. Typically, terms in these sentences are resources identified by Uniform Resource Identifiers (URIs). URIs are global identifiers for items of interest (called resources ) in the information space of the Web. Collections of RDF triples constitute an RDF graph.

Many requirements outlined as crucial for CPR systems are addressed by using RDF as a data format for patient record content. In particular, our ability to link with other clinical records can be facilitated when RDF is used in this way. Use of URIs as syntax for the names of concepts in RDF graphs is the primary reason for this. The meaning of terms used in a patient record (as well as the patient record itself or some part of it) can be made available over the Web in a (secure) distributed fashion for on-demand retrieval.

A judicious application of Semantic Web technologies can also lead to faster movement of innovation from the research laboratory to the clinic or hospital. In particular, it was envisioned that use of these technologies would improve productivity of research, help raise quality of healthcare, and enable scientists to formulate new hypotheses, inspiring research based on clinical experience.

If some or all the variables specified are in electronic format, sources must be identified, and a query made for patients in the study to extract values for the variables. This often time-consuming step is facilitated by three factors. First, at the time the information system is created, procedures can be built in to ease extracting, formatting, and exporting values for variables. This is particularly feasible in a so-called metadata-driven system, in which “data about data” drives not only the data entry process but the data extraction process as well. It is also particularly feasible for relational databases that are electronically linked (e.g., portals) to the analysis system. Second, “standard” groups of core data elements can be identified that form the basis for at least a major portion of the variables needed for most studies, and these may be part of prospectively maintained registries for quality reporting. The advantage of this strategy is that queries can be assembled carefully and refined over time. Third, successful, accurate ad hoc queries can be stored so that when the same variables are again specified, these queries can be reused.

Often more than one electronic data source must be used. In this case, values for variables in common may need to be adjudicated if they do not match in value, definition, granularity, or in variable nomenclature. Ultimately, unique variables must be joined into a common database.

Even if the majority of information is available electronically, there are nearly always some variables new to the study that must be gathered from paper records or free text in the EMR if a CPR is not available. A more arduous process must be put into place for extracting data from original documents by natural language processing. A precise methodology is necessary for assembling information to prevent repetitious handling of both the patient’s record and the extracted information, as well as to ensure complete and accurate data retrieval while preserving patients’ privacy and confidentiality. ^c

All information should be recorded in clearly defined, objective terms. There may be a preference for using descriptive terms that have been clearly defined (e.g., absent, trivial, mild, moderate, severe). Alternatively, numeric coding may be used, ^, with each numeral clearly defined. Pedal pulses, for example, may be recorded as 0, 1, 2, 3, or 4, with 4 indicating normal. Either method is equally rigorous as long as values are picked from a controlled vocabulary with clear definitions.

What one must avoid is an uncontrolled approach to entry of data in a spreadsheet ( Table 7.3 A). Each column contains data in multiple formats with a mix of alphabetical and numeric data, units associated with some numbers (differing units at that!), dates in several formats, anomalous dates in which day and month could be reversed, and different expressions of a quantitative variable. Keep in mind the data scientist or statistician who will be analyzing the data. In the end, they will need the data as it appears in Table 7.3 B. What is needed is a data entry form that has controlled data entry, in this case with separation of numerical values from a column of units of measure and non-ambiguous dates, and a data dictionary.

Accuracy of data entry is improved by recording only primary information (e.g., date of birth, date of operation) and not indices derived or calculated from them (e.g., patient age at operation, body surface area). Such indices can later be calculated quickly and reproducibly by computer.

A key concept is to record core data elements that can be logically combined in multiple ways to form derived variables (see Box 7.6 ). For example, for analysis one may want to use only the variable “current smoker.” If this were the variable gathered primarily, one would be unable later to derive other data about smoking, such as pack-years, duration of smoking, or when a previous smoker quit. Core data elements would instead relate to dates of smoking and intensity, from which all others could be derived.

The process of gathering data is the most time-consuming step in a study. It is not unusual for it to consume months or years of work. Even if electronic sources of information are used, if values for variables were not entered at the point of care, the expense can be enormous. This is why the CPR, as the repository of all patient data and patient care workflow, is essential to increase the efficiency of clinical research.

For many institutions, identifying patients using a quality assurance registry for core data elements (see Chapter 8 , Quality Assurance) and extracting more detailed ancillary data for a specific study are the most cost-effective methods for clinical research. Although review of the medical record in this way is often considered a thankless chore, it has the benefit of an investigator gaining valuable in-depth insight into the patient cohort that is not captured by typical “case report forms” or routine registry capture. Indeed, the clinical importance of the research and the clinical inferences and practical recommendations coming from the research are often greatly enhanced by careful review of all or at least a substantial sampling of the medical records.

Active follow-up means that patients or their families are contacted directly by mailed questionnaire, telephone, or electronic means (e-mail, Internet). Active follow-up is essential for discovering time-related events such as reinterventions and longitudinal clinical condition such as periodic echocardiographic assessment, perhaps with the exception of vital status, which may be available from government sources. Active follow-up data, and particularly the date of last active follow-up, must be kept separate from any augmentation of these data from passive sources. If a patient is found to have died, nearest relatives are contacted in a sensitive, sympathetic fashion to document the circumstances of death and ascertain all other pertinent cardiac events that occurred between the date of last active contact and death.

There are two general methods of active follow-up: anniversary and cross-sectional.

Anniversary method.

In the anniversary method, patients are contacted yearly on the anniversary of their surgery or entry into the study (or periodically if not yearly). This method is ideal for sampling the time-varying condition of the patient (e.g., functional status, freedom from angina, health-related quality of life, growth and developmental patterns). It has the added advantage of maintaining yearly contact with the patient or patient’s family, an important consideration in a mobile society. It has also been demonstrated that nonlethal morbid events such as thromboembolism and hemorrhage after heart valve replacement are forgotten unless there is at least yearly contact. Yearly active contact also makes it more likely patients will report events to their physicians during the course of the year.

For practical purposes, “anniversary” follow-up may be batched, resulting in a graph depicting completeness of follow-up that consists of a series of stair steps ( Fig. 7.4 A). In that case, one may wish to truncate follow-up for a given patient at exactly his or her anniversary date of surgery, even if events have happened in the interval beyond their anniversary date.

Completeness of follow-up event chart. In (A) representing anniversary follow-up, blue circles represent patients alive at last follow-up and X’s deaths plotted at the time of death on the vertical axis. The stairstep of living patients on the diagonal represents year-end assessment, and thus the blue circles are stair stepped across each year. The scattered blue circles represent transplant patients transferred to another center and lost to follow-up for the transplanting center. In (B), cross-sectional follow-up has been performed, so patients are likely to be depicted along the diagonal line than for anniversary follow-up at year end. Patients would be lined up on the diagonal if they were contacted on the exact anniversary of their surgery. In (C), a group of patients who have been followed both cross-sectionally and at anniversary at different times, showing stripes of incomplete follow-up. In (D), this same group of patients has been followed with a concerted follow-up effort, not only for vital status but for reoperations as well, something that cannot be assessed easily in passive follow-up. The blue circles lined up along 0 on the vertical axis represent patients who have not been followed after hospital discharge.

If vital status is the only outcome of interest, date of death may be obtainable from government vital statistics offices or a death registry. In passive follow-up, only death and date of death (which may be approximate) may be identified, not whether each individual in the study is alive or dead at the time of inquiry. Usually there is a lag between death and reporting, so methods must be employed to determine the status of living patients at any given time. It is important to remember that nonfatal events cannot be determined by passive follow-up. If passive follow-up is used to supplement active follow-up, a separate date for end of active follow-up must be retained for analysis of nonfatal events. Passive follow-up can include use of office visits or hospital admissions as the sole source of follow-up. This is opportunistic and is non-systematic follow-up. Non-systematic follow-up captures only some of the events and some of the potential follow-up, so analyses of time-related events are distorted. Particular care must be taken if for a clinical trial with yearly follow-up, it is required that deaths be reported within a short interval, such as 48 hours. This will result for the last year of follow-up numerators (deaths) with no denominators. This is avoided by truncating data for analysis after the preceding systematic follow-up.

In performing active follow-up, every effort must be made to contact every patient in as short a time as possible. Special assistance may be required to achieve a high level of follow-up under these circumstances. In the past, we advised using cross-reference indices to former neighbors or contact of relatives and former physicians, churches, and other agencies. However, in the United States this is now prohibited.

There is no perfect way to describe and quantify completeness of follow-up. Fig. 7.4 illustrates one way of visually assessing completeness of follow-up for each patient. In Fig. 7.4 C, note that many patients are at the bottom of the graph, indicating that no follow-up has started. Other incompletely followed patients appear below the upper triangle of completely followed patients. Lack of follow-up for nonfatal events requires that these unfollowed patients be followed cross-sectionally, as shown in Fig. 7.4 D.

Grunkemeier and Starr have described a patient-year method for estimating goodness of follow-up based on observed versus potential follow-up duration. For each patient, the duration of potential follow-up is computed (this is the interval from study entry, such as surgery, until death or, for the patient still alive at follow-up, the common closing date or response date, anniversary date, or analysis date, depending on the type of follow-up study performed). The measure of completeness of follow-up is the ratio of total observed follow-up duration to total potential follow-up duration.

In highly lethal diseases, Korn suggests a different definition of potential follow-up, namely total patient years or median follow-up as if no events occurred. ^, This is important in a setting where one has complete follow-up over 20 years, but median follow-up is only 1.5 years because of rapid demise of patients. It obviates the reader’s reaction that there is little follow-up or very incomplete follow-up, or that the therapy has only recently been introduced, all of which could be alternative reasons for short median follow-up.

Although follow-up for clinical events has dominated past cardiac surgery studies, increasingly, longitudinal data are important outcomes. These include continuous data such as valve gradient, binary data such as recurrence of arrhythmias, and ordinal data, such as grade of valvular regurgitation. Typically, longitudinal data represent a snapshot at various time intervals. But the assessments are often made at somewhat irregular times, for a different number of measurements per patient depending on whether the operation occurred recently or many years ago.

At times investigators collect and assess values only at last follow up. Such data are uninterpretable because the interval from surgery to last follow-up differs from patient to patient. Others set thresholds and when exceeded at the time of an assessment, analyze the data as if they were an event. This has several drawbacks: (1) it is uncertain when the value exceeded the threshold; all one knows is it had not at the previous assessment; (2) longitudinal data are rarely static but vary from measurement to measurement, making the threshold a moving target; and (3) a lot of detailed data are lost. As will be seen under “Longitudinal Outcomes” in Section IV, good methods are available today for analyzing longitudinal data.

When collecting longitudinal data it may seem logical to string the values in a single row on a spreadsheet, as seen in Table 7.4 A. However, this creates variable length records that are difficult to analyze. Instead, collect these repeated measures in what is known as “long format,” illustrated in Table 7.4 B. Doing so results in many records per patient ( Table 7.5 ), but the format, known as many-to-one, is ideal for analysis. Tables of such outcomes can be linked to the “one” static record of patient demographics, comorbidities, and surgical procedure in a relational database model.

The follow-up instrument may be a simple questionnaire mailed to the patient (with one or two remailings followed by telephone calls to non-responders). If so, it is wise to not exceed a single sheet in length, relying on the telephone or personal contact to obtain more details if events have occurred. Alternatively, the inquiry may be completed by telephone, using well-trained individuals, a script, and a form that is filled out during the conversation, by electronic means coupled with the patient’s medical record, or by other electronic mechanisms.

Patients rarely resent being followed up; to the contrary, periodic follow-up is useful not only in detecting medical trends in individual patients who may need attention, but also in generating good will between the patient and the medical system.

After intervals and indicator variables are created, data screening is performed. If negative intervals are found, dates or times need to be investigated and corrected in original sources. Impossible combinations of variables may be found, such as a “normal” aortic valve said to have a 100-mmHg peak gradient and valve area less than 1 cm ² . Parent–child relations are verified, particularly if the database has inadvertently overlooked constraining such relations. ^d For example, specifying an aortic valve prosthesis should be a child variable of a parent variable for “aortic valve replacement.” Inadvertent redundancies must be resolved if they disagree.

Inconsistencies in applying definitions from the data dictionary are reported to the investigation team for resolution, and the iterative process is repeated. This process is often discouraging to the novice investigator who assumes that all data (particularly those personally extracted) are flawless.

Just as important as improving accuracy of the data set is evaluating the quality of each variable by this screening and scrubbing process. One may find that information is too often unavailable in medical records to trust the variable, and it is dropped. One may also find that interpretation of the clinical condition has been so variable, such as heart failure, that the values gathered are not reproducible. Either a better surrogate has to be found or the data element dropped from further consideration.

In times past, continuous-variable data screening has consisted of producing summary tables of means, standard deviation, medians, minimum, maximum, selected percentiles, and number of missing values; quantitative variable data screening has consisted of tables of values and missing data counts. We have found these of limited value in discovering “warts” in the data. What we now do is produce pages of “postage stamp” scattergrams of every continuous variable that includes all data points, usually plotted against date of operation, and rug marks for missing values ( Fig. 7.5 ). These may reveal “outliers” that may be data entry errors, floor or ceiling effects, or an admixture of different units of measurement that need to be normalized to one measurement scale. For binary variables, we recommend two types of “postage stamp” pages: one of actual count ( Fig. 7.6 A) and a second one of percentage ( Fig. 7.6 B). These may show missing data, gaps in the data Fig. 7.6 C), different patterns of data collection, and many other anomalies that are not evident if just numbers and percentages are shown in a table. Ordinal data are displayed as stacked plots ( Fig. 7.7 ).

Sample of “postage stamp” plots of raw continuous variables as a function of date of surgery (here the number of years after time zero of surgery). The “rug marks” (short vertical lines on horizontal axis” represent missing values. The graphs demonstrate “outliers” such as BMI, BUN, LV mass, and LV mass index that should be investigated (they may be due to differences in measurement units such as centimeters and inches, kilograms and pounds, SI versus common units, coding errors, etc.). In addition, this kind of plot can reveal “block missingness” for lab measurements that did not exist until well into the study or variables not collected before a certain time. Ceiling or floor effects may be evident.

Sample of “postage stamp” plots of raw binary variables as a function of date of surgery (here the number of years after time zero of surgery). Blue portions of bars indicate the variable identified by “presence of” the title of the chart, and reddish areas represent absence of the variable. Gray portions of the bars represent missing data. In (A), actual frequency of values for variables are given, which reflects changing denominator trends over time. In (B), the data are presented as percent. Both depictions are useful. Note that if a variable is spuriously absent, such as with Native AV, the incomplete data require investigation. Some variables could abruptly change in magnitude at some point in time, which may be due to change in their definition or method of measurement. Some variables like robotic approaches or TAVR only appear when they are introduced; in that case, rather than missing values, the variable should be set to zero.

Sample of “postage stamp” plots of raw ordinal variables as a function of date of surgery and presented as stacked plots. These show the distribution of grade of mitral valve regurgitation in raw numbers (left) and as percentages (right). Note that the gray portion of bars, representing missing data, decreases over time.

In addition to these individual data checks, we routinely plot height versus weight to find “skinny giants” or “fat midgets,” along with graphs of other highly correlated variables. These drive us back to data sources to determine what the problem is and correct it, or if the cause seems to be an error in clinical recording, setting the value to missing.

It is important that corrections of errors discovered in the data anywhere along the way be fed back to the original database and the data re-exported. In this way, database quality will be constantly improved. Ideally, the change in the primary data will be documented by an audit trail.

In addition to missing values from parent–child relationships, in any study there are likely to be values for some variables that have not been recorded. Most statistical procedures eliminate entire observations (e.g., patients) for which any data requested for analysis are missing. In medical data analysis, however, one is more likely to introduce bias by eliminating all data on an entire patient than by substituting a value for the missing data that can be shown not to importantly bias the analysis. The process is called missing value imputation .

Although the literature on managing missing data is extensive, much of it is directed toward survey investigations in which entire survey instruments have not been returned. The general directive for such data is to eliminate records for nonresponders. In clinical research, missing data are most commonly sporadic or systematic (block missing) for some specific time segment (e.g., missing magnetic resonance imaging data before the introduction of that technology). These common types of missing data should be managed in a different way from those of surveys.

Once the aforementioned steps have been achieved, often iteratively, the result is a final data set in the format needed for analysis. However, one further step remains: organizing variables deemed suitable for analysis in a medically meaningful way. The reason for this is the importance we place on informed and supervised data analysis . Those analyzing the data must “know the data” just as the investigator knows the data. Not every variable has equal importance for analysis. For example, quantitative ejection fraction is “better data” than a qualitative assessment of left ventricular function on a coarsely graded scale; creatinine level at surgery contains more data than a diagnosis of renal failure; individual components of cusp morphology in atrioventricular septal defect contain higher information content than Rastelli type.

Not every variable is of equal reliability, but medical information tends to be redundant, so more reliable surrogates should be sought and analyzed. Many variables are highly correlated and may be of equal reliability, such as height, weight, body surface area, and body mass index (indeed, the latter two are calculated from the former two). Therefore, if such variables are equally associated with an outcome, and are naturally colinear, one may arbitrarily select the most reliable or easily measured representative of that concept (in this example, the concept of body size).

The data management team, in collaboration with the investigator, must then compile a final list of analyzable variables . These should be grouped in a medically meaningful fashion that aids data analysis. A suggested grouping might be as follows, although it will vary from study to study:

• •

  Demographics (age, sex, body size, social determinants of health)

• •

  Symptoms (functional status, angina class)

• •

  Ventricular function (ejection fraction, number of previous MIs, interval from last MI to surgery)

• •

  Pathophysiology and etiology (grade of mitral valve regurgitation, etiology of valvular regurgitation)

• •

  Coronary artery anatomy and disease (degree of left main disease and that of each coronary system, dominance)

• •

  Other cardiac comorbidity (previous cardiac operations, atrial fibrillation)

• •

  Noncardiac comorbidity (smoking history, creatinine, pulmonary disease, diabetes, albumin level, organized by organ system)

• •

  Preoperative management (preoperative temporary mechanical circulatory support for hemodynamic instability, intravenous nitroglycerin for unstable angina)

• •

  Cardiac procedure (CABG, aortic valve replacement, mitral valve repair)

• •

  Support techniques (duration of aortic clamping, use of warm substrate-enhanced induction cardioplegia, duration of circulatory arrest, duration of cardiopulmonary bypass)

• •

  Experience (date of operation, surgeon)

• •

  Outcome, in-hospital events (occurrence of various complications, hospital death, length of postoperative stay)

• •

  Outcome, time-related events (all-cause mortality, interval from surgery to death or censoring)

• •

  Longitudinal data (echocardiographic findings after valve operations)

• •

  Missing value indicator variables

• •

  Interaction terms (organized according to previous schema)

A practical way to implement this organizational structure is to isolate programming code in the form of a computer macro that contains the list of available variables for analysis and a place for those analyzing the data to insert code for imputing missing values, transforming the scale of variables, forming additional indicator variables, and performing other useful data manipulations. This strategy guards against human error in data analysis by isolating to a single location all data manipulation used in the analysis.

Because both calculators and computers express numbers to many digits ( Box 7.7 .), it is necessary to know a set of rules for compaction and expression (display) of numeric data. The format in which a numeric value is expressed has implications. The number 493, for example, implies that the truth is somewhere between 492.5 and 493.5 (accuracy), and that the scatter in repeated measurements of the number (precision) is no greater than that explicitly expressed ( Box 7.8 ). The number 492.8 implies that the truth is somewhere between 492.75 and 492.85, and 492.76 implies that the truth is somewhere between 492.755 and 492.765. This last numeral to the right (right-most digit) explicitly indicates that the accuracy is much greater and the precision much less than when the number is 493 or 490.

In computation and computer storage, all available digits of numbers displayed or recorded by measuring devices should be retained. It is only at the last step of numeric presentation that numbers are rounded ( Box 7.9 ). In presenting numeric information, numbers should be rounded in such a way as to reflect their precision or reproducibility, although consistency within tables is also important. ^,

Numbers are often presented in tabular form that indicates distribution of data between the extremes of a continuous variable (e.g., patient age). Such tables should be prepared so that positioning of any point along the continuous variable can be unambiguously determined. In this text, intervals between extremes of a continuous variable are indicated by symbols of inequality (see Table 7.5 ). This method of presentation of tabular information is mathematically conventional ( Box 7.10 ) but not conventional for medical publications, where ambiguity often abounds.

The other broad category of variables is continuous, for which each patient in a study (sample) may have a different value (e.g., age, weight, ejection fraction). Thus, the raw data are rarely published, because each patient or subject in a study is likely to be unique in regard to continuous variables, making any tabular presentation unwieldy unless the number of patients and number of variables are small. Summarizing statements may be made of the raw data by one of several techniques.

A commonly used summarization of raw data is a simple table with patients grouped into “nice” ordered categories. A histogram is a plot of such a table ( Fig. 7.8 A). Another method of constructing a simple table is to sort patients into several groups of equal number, even if the width of the range of values in each group is different. Because the number of such groups was originally 10, these are called decile tables .

Yet another alternative is to divide patients into percentiles , stating the value of the variable at these percentiles as follows. Patients or subjects are first sorted by (generally) increasing magnitude of the variable under consideration (e.g., by increasing age). Then the number (or more commonly the proportion) of patients with values less than or equal to each value is calculated. For example, if there are 21 patients and each is a different age at operation, patients are first sorted from youngest to oldest. No patient is younger that the youngest one (0/21, 0%, or minimum); 1/21 are as young or younger than the youngest (4.8%), and for these data, this is also the 5th percentile; 2/21 (9.5%) are as young or younger than the second youngest patient, and this is also the 10th percentile. The middle value of age, that of the 11th patient in this list, is called the median or 50th percentile . All (21/21, 100%) are as young or younger than the oldest. A cumulative distribution plot , produced easily by computer but laboriously by hand, presents all the raw data in this percentile format ( Fig. 7.8 B).

Alternatively (and more commonly), a value is found below which a stated proportion of patients have that value or a lesser one (100 times that proportion is the percentile). For example, the median is the 50th percentile. This means that half the patients have a value for the continuous variable below the median, and half have values greater. For consistency, one might also state the 15th and 85th percentiles, as they correspond to 70% confidence limits (CLs; see “ Confidence Limits [Intervals] ” in Section IV). More commonly, 25th and 75th percentiles (quartiles) or 10th and 90th percentiles are used, which summarize the middle 50% of data.

This method of summarizing data is called nonparametric (see Box 7.12 ). Beyond such simple counting (percentages and percentiles), more abstract methods are often brought into play to describe continuous data. The methods have in common a process whereby raw data on a sample of patients are used to estimate values of parameters of mathematical equations. The most familiar of these is the mean , which is estimated as the summation of all values of the continuous variable (e.g., age, pulmonary artery pressure) divided by the number of people or observations ( n ). The rationale for using the mean is that it provides an estimate of the central tendency of the data and a characteristic of the population studied. If the data are distributed perfectly symmetrically in the form of a bell-shaped curve, the mean is exactly at the midpoint of the data range (see Fig. 7.8 A). It is also the most frequently occurring number ( mode ), with half the patients above it and half below ( median ) (see Fig. 7.8 B).

Distribution of a continuous variable, age at operation. (A) Histogram of age at operation of 102 patients undergoing coronary artery bypass grafting. Approximately 30% were 50 to 55 years of age at operation, 25% were age 55 to 60, and lesser percentages of patients were older or younger. (B) Cumulative distribution plot of age at operation in these 102 patients. Vertical axis shows percentage of patients coming to operation at or younger than any given age on horizontal axis. It also gives directly the percentile of patients coming to operation by a given age: Median is the 50th percentile. S shape of this particular plot suggests a normal distribution; any other shape would suggest a different distribution.

The derivation of averages, or means, was begun by astronomers centuries ago. They thought that the scatter in their data was from observational error or imprecision, and they used means, or averages, in an attempt to obtain true values (accuracy). Later, Gauss discussed and described the symmetric normal distribution curve , which actually was described earlier by DeMoivre ( Box 7.13 ).

The mean is the easiest statistic to calculate. Unfortunately, it is not a robust measure of central tendency. If many infants and only one or two adults are in a study, average age is greatly exaggerated by the few adults. A more robust measure of central tendency is the median. Whether or not the sample data are distributed in a Gaussian-type bell-shaped curve (see Fig. 7.9 and Box 7.13 ) may be tested by such statistics as the Shapiro-Wilk W statistic for small n (e.g., 50 or less) and the Kolmogorov-Smirnov D statistic for larger samples. The skewness of the data (rightward or leftward asymmetric tail) and their kurtosis (unusual peakedness of the distribution of values) are also tested.

Gaussian bell-shaped distribution curve for age. (A) Distribution curve (smooth line) fitted to data summarized in underlying bar graph (see also Fig. 7.8 A). (B) Distribution with mean ±1 and ±2 standard deviations (SD) marked by vertical lines. Point of inflection of the curve from concave to convex is 1 SD.

Thus, in addition to an estimation of the population mean, some measure of dispersion (variance, spread, scatter) of values is needed. One such measure is the standard deviation , the name of the second parameter of the Gaussian distribution equation (see Box 7.13 ). It refers to variability from subject to subject or variability of individuals within the sample and is used to determine whether an individual is “within limits of normal.” Standard deviation is necessary for comparison statistics. For example, an individual’s standard deviation from the mean regarding a particular measured variable (commonly called z ) is often useful. This is calculated from the difference between the measurement for the individual and the mean normal value divided by the standard deviation. A z may be negative or positive and has no units ^e (see “ Standardization of Dimensions ” under “ Dimensions of Normal Cardiac and Great Artery Pathways ” in Chapter 1 ).

However, if the distribution of data values does not conform to the normal distribution, then reporting two model parameters of the normal distribution, namely mean and standard deviation, represents a mismatch between the data and the model of the data, called model misspecification. Thus, for a continuous variable that is strictly positive (and nearly all of them in medicine have only positive values), if the standard deviation is larger than the mean or even close to it, the data are skewed ( Table 7.6 and Fig. 7.10 ), and median and selected percentiles must be provided instead to accurately summarize the distribution of data.

Illustration of a normally distributed variable and three skewed distributions. In (A), probability density functions (histograms and a smooth curve) are skewed “to the right” (long tail to the right). This is typical of many medical variables whose value cannot be less than zero. The long right tail often indicates pathologic values. In (B), cumulative distribution functions that correspond to the area beneath density functions shown in A . “Normally” distributed values take on an S-shaped curve, which is not so for the skewed variables. Parametric statistics such as the mean and standard deviation are appropriate for the more-or-less normal bell-shaped curves, but nonparametric statistics are usually needed for the skewed distributions. BUN , blood urea nitrogen; ICU , intensive care unit.

Standard error is a measure of the reliability with which the population mean is estimated from the sample mean, and it is needed for comparing one group with another. It is more appropriately (but infrequently) called the standard deviation of the mean and is obtained simply by dividing the standard deviation by the square root of n (see Box 7.13 ).

Other methods are available for summarizing skewed data. One is to resort to a purely nonparametric (i.e., without equations, coefficients) description (e.g., using the median and its various percentiles). ^, Another is to transform the data into a more normally distributed scale. For example, a logarithmic transformation is often useful; the resultant mean is called the geometric mean .

The questions “What is the risk of repair of postinfarction VSD in general?” and “Is risk with the method of repair I used higher or lower than that with the method another surgeon is using?” are similar to questions put to Galileo about the nature of chance, particularly games of chance, by 17th-century gamblers. ^, From those questions emerged the Laws of Chance, now known as the theory of probability . These laws are believed to apply to all things that can have more than one possible result. Many scientists believe that all natural phenomena, including those of the physical world, behave in accordance with the theory of probability. ^, Events and phenomena of cardiac surgery behave in accordance with this theory.

Galileo showed that there is variability in sample point estimates. To illustrate, if the risk of death in the entire population of patients undergoing repair of postinfarction VSD by a given method is 33%, and samples of three patients are taken repeatedly, 0 deaths among the three would be experienced in 30% of samples, 1 death in 44% of samples, 2 deaths in 22% of samples, and 3 deaths in 4% of samples. In larger samples, results are less variable. For example, with samples of size 300, although the number of deaths experienced may still be quite variable, the proportion dying will be 30% to 36% in 70% of samples taken.

Because of this random variability in the sample estimates of risk, it is impossible to estimate the population parameter (see Box 7.11 ) with certainty (i.e., to know the risk in the entire population) from sample information. However, the pattern of variability in repeated sampling is well understood, and in most situations, it is possible to derive a formula to calculate the range of values that would contain the parameter for a specified percentage (e.g., 70%) of samples taken.

Users of CLs should be aware that this range of values for all proportions except 0.5 (50%) is asymmetric, in contrast to standard deviations, which are symmetric. Thus, we must report both the point estimate (probability) and lower and upper CLs.

As the sample size increases and more information becomes available, width of the confidence interval decreases (i.e., a more precise estimate is obtained). With a more precise estimate, the investigator is less uncertain where the population parameter lies, or in other words, what the “true” risk is. With a less precise estimate, the investigator is more uncertain.

A number of methods have been developed to calculate CLs for proportions. ^, Bootstrapping ^f is a generalized method for obtaining CLs for any statistic. The original sample of data is randomly sampled in such a way that the patient can be sampled again (sampled with replacement) to form a data set equal in size to the original. Because of replacement, some patients will appear more than once in this bootstrap sample, and others will not appear at all. The point estimate (e.g., hospital mortality) is estimated in this sample. Then another sample is drawn in the same fashion, and this process is repeated as many as 1000 times. All the point estimates from each sample are sorted from smallest to largest, as in forming a cumulative frequency distribution (see “ Descriptive Data Exploration ” in Section III). The “best” estimate of the point estimate is the median value (50% above and 50% below). If 70% CLs are desired, then the 15th percentile is the lower CL and the 85th percentile is the upper limit (if 68.3% limits are desired, the numbers would be approximately the 16th and 84th percentiles). Approximating formulae are used in most statistical packages.

Any desired CL can be derived, such as 50%, 70%, 90%, 95%, or 97.5%. Choice of CLs to be expressed (called the confidence coefficient ) depends on (1) use to be made of them, (2) consistency, or (3) convention, in that order of preference.

Most often in cardiac surgery, CLs are used as scanning tools to aid predictions and comparisons, either of proportions or time-related depictions (see “ Scanning Tool ” later in this section). If great certainty is desired in the inference that there is a difference between two proportions of time-related depictions, 95% confidence intervals may be chosen for the comparisons. If only moderate certainty is required that the evident difference is a true difference and would be found in larger samples, 50% confidence intervals might be chosen.

Most situations in cardiac surgery seem to lie somewhere between these extremes, so use of 70% CLs for most comparisons is reasonable. The interval is relatively narrow (specific), and although it is reasonably certain that truth lies within the CLs, there is a 15% chance it will be higher and a 15% chance it will be lower.

Seventy-percent CLs (actually 68.3%) are equivalent to 1 standard error (SE), and 95% CLs are consistent with 2 SEs. For consistency, if other numeric estimates are presented to 1 SE, 70% CLs should be used, and if 2 SEs are presented, 95% CLs should be used. We emphasize consistency because we believe surgeons should become familiar with using CLs as a scanning tool; to use a tool effectively, it is helpful to be consistent among all measures of uncertainty. Conventionally, many statisticians use 95% CLs, even in the context of using 1 SE for most everything else, and 50% limits for nonparametric statistics. This makes no sense and is simply a habit, not a product of reflective thinking about the inferences or about consistency.

In a numeric presentation of differences, such as difference in survival curves (see Box 7.3 ), 90% CLs are equivalent in comparative inference to individual 70% CLs, a largely empirical finding. The reason is that a one-sided confidence interval of a difference between two estimates is narrower than the sum of the 70% upper and lower CLs that just touch. This narrowness is compensated for by use of somewhat wider CLs (90%) of the difference.

Overlapping or nonoverlapping of CLs around two or more point estimates can be used as a simple and intuitive scanning method for determining whether the difference in point estimates is unlikely to be due to chance alone. They delimit the effect, and because they are accompanied by the magnitude of the effect, there is no confusion between statistical significance and magnitude of the effect, as there may be if P values are used (see “ P -Values ” later in this section). When CLs are not overlapping, the difference is unlikely to be due to chance alone.

Because “nonoverlapping CLs suggest with a stated degree of uncertainty that a difference exists” is cumbersome, the phrase evident difference may be used to express the same idea ( Appendix 7B ). Nonoverlapping CLs are easily visualized in a nomogram in which the CLs are displayed around the point estimate expressing the association between variables. Within this context, it can be said with a stated degree of uncertainty that the effect of the independent variable compared with a baseline value becomes evident at the point at which the CLs just separate. However, in contrast to evident differences in a contingency table, this point is not easily seen in a nomogram, and it does not appear in an equation. The point at which evident differences appear in equations can, however, be calculated mathematically (see Appendix 7B ).

We stress that comparing CLs in this way is a scanning tool . The classic method using P values involves computing the difference between the two proportions and testing the hypothesis that the difference is zero. Experience with scanning and P value methods has taught that when the lower 70% CL of one estimate just touches the upper 70% confidence of the other, the P value for the difference is between.08 and.1; when similar 95% CLs just touch, the P value is about.01.

In the context of hypothesis (or significance) testing, the P value is the probability of observing the data we have, or something even more extreme, if a so-called null hypothesis is true ( Box 7.15 ). Or, as stated by the American Statistical Association (ASA), “Informally, a P value is the probability under a specified statistical model that a statistical summary of the data (e.g., the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.” However, if you must have a formal definition, three of them are presented in commentaries within the ASA white paper by Deborah Mayo, Michael Lavine, Joseph Horowitz, and Valen Johnson.

Historically, hypothesis testing is a formal expression of English common law. The null hypothesis represents “innocent until proven guilty beyond a reasonable doubt.” Clearly, two injustices can occur: a guilty person can go free or an innocent person can be convicted. These possibilities are termed type I error and type II error , respectively (see Box 7.15 ). Evidence marshaled against the null hypothesis is called a test statistic , which is based on the data themselves (the exhibits) and n. The probability of guilt (reasonable doubt) is quantified by the P value or its inverse, the odds [(1/ P ) − 1] (see Box 7.3 ).

Had the originators been raised under a different judicial system, perhaps a different pattern for testing hypotheses might have arisen. Specifically, the system does not judge how innocent a person is (the “alternative hypothesis”; see Box 7.15 ), nor does it test for equivalence, a very important matter for comparing pharmaceuticals and even alternative surgical therapies.

Some statisticians believe that hypothesis or significance testing and interpretation of the P value by this system of justice is too artificial and misses important information. For example, it is sobering to demonstrate the distribution of P values by bootstrap sampling. Furthermore, the magnitude of the P value is dependent on two factors: magnitude of difference and sample size. These individuals would prefer that P values be interpreted simply as “degree of evidence,” “degree of surprise,” or “degree of belief.” We agree with these ideas and suggest that rather than using P values for judging guilt or innocence (accepting or rejecting the null hypothesis), the P value itself should be reported as degree of evidence. It is worthwhile considering the conclusion of the ASA white paper: “Good statistical practice, as an essential component of good scientific practice, emphasizes principles of good study design and conduct, a variety of numerical and graphical summaries of data, understanding of the phenomenon under study, interpretation of results in context, complete reporting and proper logical and quantitative understanding of what data summaries mean. No single index should substitute for scientific reasoning.”

All methods for calculating P values have in common one or more point estimates, some measure of variability for each, some comparison statistic related to the point estimates (e.g., the difference or a ratio), an estimate of the variability of the comparison statistic, and size of the groups.

The test to be used is selected. This must be appropriate for the comparison. It is crucial that a biostatistician familiar with the data and desired comparison be the one to select this test and interpret the results. In general terms, this demands that a specific distribution of the difference or ratio be selected. From the difference or ratio, some measure of its variability ^, and n , a number, is computed for the particular distribution selected, called the test statistic (see Box 7.15 ). There are a number of test statistics, which means there are a number of prescribed, defined, specific methods (tests) for calculating the test statistic. The statistician selects the test statistic to be used on the basis of the fit of the data to the assumptions underlying the test.

The magnitude of the computed test statistic among the hypothetically determined distribution of values for the test chosen is determined. The area under the distribution curve (proportion of the total area) occupied by more extreme values of the test statistic is the P value , a number ranging from 0 to 1.

In the case of many test statistics, a family of distribution curves exists, and to determine the P values, one of these must be selected. The selection is based, more or less, on the sample size ( n ). By “more or less,” we mean that some information content in the n may already have been “used up” in other calculations in the process and may not be available for computation of the P value. What is left, called degrees of freedom , determines the distribution curve selected.

The phrases one-tailed P value and two-tailed P value are commonly used. Which is appropriate depends on the research hypothesis being tested. When the hypothesis relates to differences in either direction (“different from zero”), a two-tailed P value is used; when it relates to differences in only one direction (“less than,” for example), a one-tailed P value is used. A two-tailed P value is always the same as or larger than a one-tailed P value. Generally, in the work described in this book, two-tailed P values are used.

Advances in machine learning for multivariable analyses evolved rapidly beginning in the late 1990s. Freund and Schapire introduced “AdaBoost” (adaptive boosting) for classification, a precursor to the now widely used “gradient boosting” method. A remarkable feature of AdaBoost was that it required little supervision and appeared to be immune to overfitting. In many experiments it was found to outperform traditional methods, especially in challenging examples where standard model assumptions failed to hold. The mathematical explanation for AdaBoost’s success, however, seemed very mysterious: The algorithm iteratively reweighted observations fitting in each iteration a simple “weak” learner (such as a tree) with higher weights given to those data points difficult to classify. This seemed intuitive, but a rigorous explanation of what it was doing was not clear at first.

Fig. 7.11 provides an illustration of how AdaBoost uses its weights to construct a classifier. Data were simulated from the circle in a square synthetic experiment (in this experiment, we have two covariates, x ₁ and x ₂ , and the outcome is one of two classes: class one is a circle and class two is the outside of the circle inside a square). Stumpy trees with a depth of 3 were used for weak learners (trees are characterized later in this section). The figure shows how AdaBoost’s weights vary with number of iterations, m. As seen, weights quickly migrate from a uniform distribution to small and large values, where large values are observed near the classification boundary (where the circle touches the square). The last panel displays test-set prediction error. Error drops substantially with only a few trees: This rapid effect is due to classifying easy cases, those away from the boundary, which make up the majority of the data. After that, AdaBoost spends its effort trying to improve classification for the hard to classify cases near the boundary, and the effect on reducing prediction error is much slower.

Circle in a square two-class synthetic experiment using AdaBoost with stumpy tree learners with depth of 3. Shown are AdaBoost weights for iterations m = 1; 2; 3; 4; 5; 6; 10; 50. Size of weights are proportional to size of points: Values near the boundary of the circle with the square are hardest to classify and receive the largest weights as m increases. Bottom right figure shows test set prediction error as a function of m (number of trees).

Many learning methods apply the principle of empirical risk minimization. In many cases empirical risk equals the training error. For example, in regression the empirical risk is equivalent to goodness of fit, which essentially amounts to measuring how well the procedure performs in fitting the data. These procedures seek to minimize empirical risk and are constructed in a manner allowing them to fit flexible models while avoiding overfitting (this latter aspect of machine learning to avoid overfitting is called regularization). Empirical risk minimization is a well-understood concept with good properties, but how then was it possible for AdaBoost to work so well? There seemed no reason to believe that AdaBoost was an empirical risk-reduction strategy.

Surprisingly, researchers were later able to show that the weighting schemed used by AdaBoost was equivalent to fitting an additive stagewise learner. In fact, the algorithm was shown to be minimizing a loss function nearly identical to that used in logistic regression. The term stagewise refers to a procedure that repeatedly refits residuals. Thus, AdaBoost repeatedly refits residuals, each time seeking to minimize a loss function and its associated empirical risk.

The insight that AdaBoost was actually performing risk minimization quickly led to a unified treatment for general loss functions applicable to many problem settings. This unified treatment is now generally referred to as gradient boosting and is taken to be the modern standard for boosting. Just like AdaBoost, the idea is to repeatedly fit the data using weak learners. Gradient boosting, however, sets about choosing the weak learner using the principle of steepest descent, thereby directly attacking the issue of empirical risk minimization head on. (See also “matching pursuit,” which is a closely related idea. ) The AdaBoost algorithm was originally proposed for classification. It constructs its classifier by using an additive expansion F M = ∑ m = 1 M c m h m , where h ₁ , h _2, …, h _M are weak learners and F _M is the combined learner used for classification. Therefore, AdaBoost adopts the view of combining weak base learners for the purpose of creating a better classifier. A closely related idea is ensemble learning, which was another intensely studied area in the early development of machine learning. Researchers found, surprisingly, that a procedure’s performance could be significantly improved by combining its runs over different subsets of the data. The resulting averaged procedure was termed an ensemble (as an example, an ensemble classifier is a collection of classifiers whose votes are averaged). One especially successful procedure is bagging, which combines bootstrapped trees. Random forests refine the idea of bagging by introducing further randomization into the tree construction. The averaged random trees result in a more accurate ensemble than bagging.

A characteristic common to the aforementioned procedures is that they can all be implemented with little supervision. Typically, only a few parameters, called tuning parameters, need to be determined, and choosing them wisely prevents overfitting. Tuning of parameters is the so-called regularization step needed when fitting a machine learner. For gradient boosting, the most important parameter is the number of iterations (in our circle in the square problem, this is the number of trees used); for random forests, they are the parameters specifying how a random tree is constructed, for example, how many random features to be selected when splitting an internal tree node or how large the terminal nodes (end of a tree) are. Importantly, a hallmark of these methods is their robustness to tuning parameters and that parameter tuning is generally easy to do in practice; typically, this is accomplished by using cross-validation. Another hallmark is their ability to automatically learn from the data and find difficult-to-grasp relationships, such as interactions and nonlinear trends without human input. For this reason, these procedures are generally referred to as machine learning methods.

Screening candidate variables has two purposes: (1) to determine whether there are sufficient data (see Box 7.4 ) to be suitable in the analysis and (2) to understand a variable in relation to other candidate variables. Because for outcome events the effective sample size for analysis is the number of events, not the number of patients, a variable may not be suitable for analysis when it represents a subgroup of patients with too few events to evaluate. This represents a limitation of the study, not of methodology. Indeed, one is generally happy with a therapy associated with few events ; however, it then makes sense that risk factors cannot be identified.

We do not screen variables to discover which ones relate individually to outcome. It is a common practice of many groups to ignore variables that are not univariably associated with outcome. However, there is a long history of occurrence of lurking variables ( Box 7.18 and Fig. 7.12 ) that are found to relate to outcome only when (1) other variables that mask their importance are accounted for in the analysis or (2) they are suitably transformed (or coupled with nonlinear rescaling of themselves), indicating a complex association with outcome. ^,

Demonstration of a lurking variable. Survival after stress testing is shown on an expanded scale and stratified according to use and nonuse of long-term aspirin therapy. (A) Risk-unadjusted survival in entire cohort. Note similarity of survival. (B) Survival in propensity-matched patients. Note dissimilarity of survival revealed when risk factors for death are balanced between groups.

(From Gum and colleagues. )

It is valuable to determine the pairwise correlation of variables. This will help one understand why many variables may be associated with outcome, but only a few are selected as risk factors. Medical data are highly redundant, sharing a great deal of information.

Continuous variables contain unique values for each patient and so are particularly valuable in analyses. For unclear reasons (statisticians uniformly decry the practice), many investigators stratify continuous variables into two or a few arbitrary categories, throwing away valuable information. This flies in the teeth of a fundamental philosophy of data analysis: continuity in nature (see “ Continuity Versus Discontinuity in Nature ” in Section I). Furthermore, to better understand the phenomenon one is studying, it is important to determine the shape of the relation of continuous variables (e.g., age, birth weight, creatinine) to outcome.

However, the scale on which a continuous variable has been measured or expressed may not coincide with a linear increase in risk. Nature does not know about man-made rulers! Therefore, the appropriate calibration of the variable to outcome must be discovered. One method to accomplish this is to examine various linearizing transformations ( Fig. 7.13 ). However, the “perfect” transformation of scale may not coincide with the best one after other factors have been considered in a multivariable model. Thus, we rely on graphical methods, as in the figure, to obtain a set of similar transformations, and then include all transformed variable candidates in the selection process to be described. A promising offshoot of nonparametric machine learning techniques, such as random forests technology, is the generation of risk-adjusted coplots and risk-adjusted partial dependency plots that can suggest the shape of the relationship of these continuous variables with risk ( Fig. 7.14 ). ^,

Calibration of 1-second forced expiratory volume (FEV ₁ ) to risk of hospital mortality. Scale of risk is given on vertical axis (akin to logit units of Fig. 7.1 ), and eight groups of equal numbers of patients according to value for FEV ₁ along horizontal axis. Their mortality, converted to the risk scale, is shown by each closed circle . (Eighth closed circle cannot be shown because there were no deaths in the eighth group with highest FEV ₁ s.) (A) Linear scale of FEV ₁ . Clearly there is a decreasing (more negative) value of risk at higher FEV ₁ (simple regression line shown, with explained scatter for these points of 80%). (B) Inverse scale of FEV ₁ . Because of the inverse transformation, lower FEV ₁ s are to right of scale, and higher FEV ₁ s to left. Risk falls from left to right, unlike in A . There is now tighter correspondence of risk to this rescaling of FEV ₁ (85% of scatter explained) than in the conventional scale of A .

(From Blackstone and Rice. )

Risk-adjusted partial dependency plot from a random forest analysis of operative mortality demonstrating the “shape” of its relationship to a number of continuous variables representing demographics and organ dysfunction.

A seminal contribution of the Framingham Study investigators was the idea that in the absence of identified mechanisms of either disease or treatment failure, useful inferences for medical decision-making, lifestyle modification, and programmatic decisions about avenues of further research can be gleaned by nonspecific risk factor identification. A direct consequence of the idea, however, is that for any set of potential variables that may be associated with outcome, there is no unique set of risk factors that constitute the best common denominators of disease or treatment failure. Therefore, different persons analyzing the same data may generate different sets of risk factors. As a consequence, multivariable identification of risk factors has become an art that depends on expert medical knowledge of the entity being studied, understanding the goals of the research, knowledge of the variables and how they may relate to the study goals as well as to one another, identification of the quality and reliability of each variable, and development of different, often sequential, analysis strategies appropriate to each research question. ^{, , ,} Not all these issues of art or expertise will disappear, but there are substantial aspects of multivariable analysis that are yielding to science.

Naftel of UAB, in an important 1994 letter to the editor of the Journal of Thoracic and Cardiovascular Surgery, addressed nine aspects of multivariable analysis that contribute to obtaining different models (sets of risk factors). He called these “steps and decisions that may influence the final equation”:

• •

  Differing statistical models. For example, if time-related events are being modeled, results using a Cox proportional hazards model (see “ Cox Proportional Hazards Regression ” later in this section) will differ from those using a multiphase nonproportional hazards model (see “ Parametric Hazard Function Regression ” later in this section).

• •

  Differing approaches to missing data (see “ Managing Missing Values ” in Section III).

• •

  Differing approaches to minimal information (see Box 7.4 ).

• •

  Differing approaches to correlated data. Variables with similar information content should be chosen for maximal insight by the clinical investigator, not necessarily the statistician.

• •

  Differing coding of data. Some may pay more attention than others to linearizing transformation of continuous variables, to whether continuous or ordinal variables should be dichotomized or in other ways collapsed, or to management of interaction (multiplicative) variables.

• •

  Differing approach to apparently incorrect data. True data outliers, handling of clearly imperfect data, improbable combinations of variables (e.g., apparently exceedingly short, very heavy patients as a result of misplaced decimal points or mixed metric and English units), and attitude toward whether or not a large sample negates errors are all decisions made during the screening process for multivariable analysis.

• •

  Differing variable selection methods and P value criteria. This area is undergoing complete change through introduction of machine learning algorithmic methods. Even with new methods, however, a criterion must be arbitrarily established to differentiate what is signal from what is noise ( P values, for example).

• •

  Differing computer resources. Although even desktop computers rival the computational capacity of large-scale computers of several decades ago, computer-intensive methods may require high-intensity parallel processing.

• •

  Differing appreciations of the science. Unless data analysts work collaboratively with the surgeon-investigator, analysis may be unrevealing. One cannot divorce the underlying clinical science from data analysis.

In all areas, new knowledge has been generated that is beginning to differentiate inadequate techniques from reasonable techniques and optimal techniques. Perhaps the more active area presently is “differing variable selection methods,” and it is an important one. Part of the challenge is that variables may be thought to be risk factors because they are associated with a small P value, and other factors may be thought not to be risk factors because of larger P values, but both opinions may be erroneous (type I and type II statistical errors, respectively; see Box 7.15 ). There is therefore a need for a method that balances these two types of error. Closely coupled with this is the need for a statistic that measures the reliability with which a risk factor has been identified. Because one is analyzing only a single set of data rather than many sets of data about the same subject, determining this reliability has been elusive. It is in the arena of machine learning that promising solutions have arisen to address this gap in knowledge.

Variable selection by bootstrapping.

Thus, there is new thinking about what risk factor identification is. In thinking anew, we leave traditional statistical methodology out of the picture, and risk factor identification becomes an attempt to find signal (risk factors) in noise (other candidates). Important advances in pure mathematics ( logical analysis ) ^, and machine learning ( algorithmic analysis ) are proving valuable for such diverse signal detection challenges as handwriting identification, genomic identification, and now risk factor identification. These techniques are evolving rapidly, and we will describe only the most basic here: bootstrap aggregation, or bagging .

Bootstrapping belongs to a class of methods that has been developed over the past 40 years. In 1983, an astonishing article entitled “Computer-Intensive Methods in Statistics” appeared in the popular scientific literature. Its authors, Persi Diaconis and Bradley Efron from Stanford University, indicated that “most statistical methods in common use today were developed between 1800 and 1930, when computation was slow and expensive. Now, computation is fast and cheap. The new methods are fantastic computational spendthrifts….The payoff for such intensive computation is freedom from two limiting factors that have dominated statistical theory since its beginnings: the assumption that the data conform to a bell-shaped curve and the need to focus on statistical measures whose theoretical properties can be analyzed mathematically.”

Efron and his group demonstrated that random sampling with replacement ^h from a data set to create a new data set, resampling to produce perhaps thousands of new data sets, and combining the information generated from these many data sets can produce robust and accurate statistics without assumptions. His group called this technique bootstrapping , after the expression “pulling yourself up by your own bootstraps,” because it reflected the fact that one could develop all the statistical testing necessary directly from the actual data simply by repeatedly sampling them (see footnote g, p. 278).

These techniques have been applied to entire analytical processes, including multivariable analysis. ^, In fact, one still has to pay attention to appropriate models, missing data, variable considerations, correlated variables, appropriate strategy, and so forth, that remain part of a disciplined, informed approach to the data. However, the variable selection process is bootstrapped.

In practice, a carefully crafted set of variables is formulated that will be subjected to simple automated variable selection, such as forward stepwise selection, whereby the most significant variables are entered one by one into a multivariable model. Specific P value criteria for entering and retaining these variables are specified. Then a random bootstrap sample of cases is selected, generally of the same sample size as the original n. A complete automated analysis is performed, and its results are stored. Then another random bootstrap set of cases is drawn from the original data set, and analysis is performed. This resampling of the original data set, followed by analysis and storage of the results, continues perhaps hundreds and even thousands of times, then the frequency of occurrence of factors identified among these many models is summarized. Frequency of occurrence generally stabilizes after about 100 bootstrap analyses. The many models are also analyzed by cluster techniques to detect closely related variables that in the final model will be represented by the most commonly occurring representative and by noting if a variable and one or more transformations of scale occur with about equal frequency, indicating a nonlinear relationship to risk. All this information is used to select variables for the final multivariable model.

Of interest, the variables identified for every bootstrap data set are usually different, a sobering revelation. However, it becomes evident that some variables are never selected or seldom selected; these constitute “noise.” Variables that appear in 50% or more of models are claimed to be reliable and are considered “signal” for inclusion in the final model.

This phenomenon is illustrated in Table 7.7 and Fig. 7.15 . Fifteen variables were selected from among many being analyzed for the late hazard phase of death following mitral valve repair or replacement for degenerative disease. In analysis of the first bootstrap sample, 8 of these 15 variables were selected (only 5 were ultimately found to be reliable risk factors). By 100 analyses, although every variable had been identified as a risk factor in at least 2 analyses, 5 variables dominated the analyses (we considered these reliable risk factors), 8 rarely appeared, and 2 appeared in 22% to 32% of analyses.

TABLE 7.7

Frequency of Occurrence (%) of Variables Selected in Bootstrap Analyses of the Late Hazard Phase of Death after Mitral Valve Repair or Replacement for Degenerative Disease

	NUMBER OF BOOTSTRAP ANALYSES
Variable	1	5	10	55	100	250	500	1000
Demography
Age	100	100	100	100	99	99	99	99
Women	0	0	0	6	7	4	3	5
Noncardiac Comorbidity
Bilirubin	0	40	20	16	12	10	10	10
BUN	100	40	60	72	76	78	77	78
Hypertension	0	0	10	6	6	5	6	6
Peripheral artery disease	0	0	0	4	2	4	3	3
Smoker	0	0	0	6	8	9	11	10
Ventricular Function
Ejection fraction	0	0	0	18	22	22	24	25
Left ventricular dysfunction (grade)	100	60	70	70	66	66	68	68
Right ventricular systolic pressure	100	20	10	10	8	8	8	8
Cardiac Morbidity
Coronary artery disease	100	100	100	96	94	92	92	91
Anterior leaflet prolapse	100	80	90	82	82	84	85	85
Preoperative Condition
NYHA class	100	20	20	30	32	34	33	36
Hematocrit	0	0	20	16	17	14	16	17
Experience
Date of operation	100	20	10	14	15	14	13	14

BUN, Blood urea nitrogen; NYHA, New York Heart Association.

Example of automated variable selection by bootstrap aggregation (bagging). Fifteen variables labeled A through O are depicted as potential predictors of death after mitral valve surgery. In column A, analyses of five bootstrap samples are shown. Tall bars indicate the variable was selected at P <.05, and gaps represent variables not selected. In all cases, variables A and D were selected, but otherwise analyses appear to be unique. Panel B shows a running average of these five analyses. Variables A, D, I, and J were selected more often than others. Panel C shows averages of 10, 50, 100, 250, and 1000 bootstrap analyses. Notice that no variable was selected 100% of the time, and all 15 were selected at one time or another. But if we consider variables appearing in 50% or more analyses as reliable risk factors, variables A, C, D, I, and J fit that criterion of “signal” and the rest are “noise.”

(From Blackstone and colleagues. )

What happens in bagging (bootstrap aggregation ) is similar to what is seen in signal averaging, such as in visual evoked potentials. Noise is canceled out, and signal amplified. In the same way, many variables appear rarely in models, but a few show up time and time again (see Fig. 7.15 ). One can therefore express the reliability of identification of a given risk factor at a selected level of statistical significance.

Bagging, although demanding a huge number of computer cycles, removes much of the human arbitrariness from multivariable analysis and provides another important statistic: a measure of reliability of each risk factor. Thus, increasingly we have been reporting not only the magnitude of the effect, its variance, and its P value, but also its bootstrap reliability. The technique appears to provide a balance between selecting risk factors that are not reliable (type I error) and overlooking variables that are reliable (type II error).

Variable selection by machine learning.

Variable selection for traditional parametric models generally relies on use of statistical significance ( P values) or ad hoc stepwise methods for reducing dimension and choosing variables. Issues with P values have been discussed earlier, and stepwise procedures are unreliable because their results are highly dependent on the order in which variables are entered or eliminated. Neither of these procedures is suitable when number of variables can be high. A regularization procedure to address this is the least absolute shrinkage and selection operator (lasso) method. In the case of regression, the lasso is fit in the same manner as least-squares but with the additional constraint that the length of the regression parameters must be constrained. The lasso measures length using absolute distance (mathematically called L ₁ -distance), which is the sum of the absolute values of the coefficients (for example, if there are two variables x ₁ and x ₂ with coefficients β ₁ and β ₂ , then the lasso penalty term is λ _l (| β ₁ | + | β ₂ |), where λ _l > 0 is the lasso regularization parameter (as will be explained, larger values of λ _l induce more penalization and therefore sparser solutions). This differs from ridge regression that constrains the length of parameters using Euclidean distance (called L ₂ -distance). Thus, in our example the ridge penalty is λ _r (| β ₁ | + | β ₂ | ), where λ _r > 0 is the ridge regularization parameter.

Ridge regression was introduced to combat instability of least squares in linear regression arising from the presence of collinearity among the covariates. By applying the ridge penalty, we can always be assured that a solution exists even in high correlation and even in the presence of many covariates. This is a nice feature of using ridge penalization; however, the problem is that this does not address the issue of variable selection. The seemingly small difference between L ₂ -regularization used by ridge versus L ₁ -regularization used by lasso has tremendous consequences for selecting variables, especially in high dimensions. Unlike ridge regression, the estimated regression model using lasso will have coefficient values that are exactly zero, whereas ridge regression is not able to set any coefficient values to zero. Thus, the lasso has the desirable property that it achieves estimation and variable selection simultaneously, the latter being achieved by having coefficients that are exactly zero and therefore of no value to the model.

How does the lasso achieve this? Consider Fig. 7.16 . Displayed is the optimization problem for the lasso in a regression problem involving three variables, labeled x, y, and z . As mentioned, lasso is a penalization problem that penalizes the least squares solution by absolute distance. The least squares solution (the usual regression solution) is the center of the ellipsoid. To solve the penalization problem, the solution is to find the point at which the ellipsoid first touches the constraint region, which is the pointy region centered at zero. The size of the constraint region depends on the lasso parameter λ _l , with larger values applying more constraint. Because of the shape of this region, it will often happen that the solution will touch one of the axes, which in this case is x . The result is that this coefficient becomes zero. In problems with many variables, the pointy nature of the constraint region will induce many variables to be zero. This leads to the so-called sparsity property of the lasso in high dimensions.

The lasso solution is the point where the ellipsoid touches the lasso constraint region | x | +| y | +| z | ≤ C .

Now consider Fig. 7.17 , which is the solution for ridge regression. The difference here is that the constraint region is a sphere centered at zero. The ridge solution is where the ellipse touches the sphere. The size of the sphere is related to the ridge parameter, which like the lasso parameter controls the amount of regularization. However, due to the round nature of the sphere there is no possibility for any coefficient to become zero like the lasso. This problem persists and becomes worse as the number of variables increases. Thus, the ridge estimator does not possess sparsity and cannot be used for variable selection like the lasso.

The ridge solution ( p = 3 dimensions) is the point where the ellipsoid centered at the ordinary least square value touches the constraint region x ² + y ² + z ² = C .

In another promising approach, machine learning technologies are being harnessed in interesting ways by either embedding traditional parametric models or extending nonparametric analytic strategies. As mentioned, results of traditional variable selection are highly dependent on the order in which variables are entered or eliminated. One can instead imagine forming thousands of bootstrap models with clusters of randomly chosen variables forced into each and aggregating the results. One can apply learning theories to model development. One can examine variable importance (often revealing that many variables actually degrade predictive power). Splitting algorithms can be averaged to reveal the most common splits ( Fig. 7.18 ).

Use of random forests for variable selection. (A) Example of a random tree. A bootstrap sample of patients from original data set is used to create a random tree. At the root node, a random set of variables is chosen to be candidates, and the most predictive variable for survival among those is identified. Node levels are numbered based on their relative distance to top of tree (i.e., 0, 1, 2). Splitting of nodes to create trees continues until terminal nodes have a few distinct events (e.g., deaths). (B) Illustration of minimal depth of a variable in a random tree from a 2000-tree forest. Highlighted are three top variables: peak V ˙ o 2 (violet) , blood urea nitrogen (BUN, aqua ), and exercise time (tan) . Depth of a node is indicated by numbers 0, 1, 2, 3-8. Minimal depths are 0, 1, 2 for exercise time, peak V ˙ o 2 , and BUN, respectively. (C) Illustration of six random trees from a 2000-tree forest. The three most important variables among these trees are color coded blue for treadmill exercise time, violet for peak V ˙ o 2 , and green for serum BUN. (D) Minimal depth (variable importance) from random survival forests analysis. Dashed blue line is threshold for filtering variables: All variables below line are predictive. Diameter of each circle is proportional to forest-averaged number of maximal subtrees for that variable.

(From Hsich and colleagues. )

In machine learning, variable selection is often performed using variable importance (VIMP), defined by how much prediction accuracy of the model depends on the information in each feature. ^{, ,} One of the most popular methods is permutation importance, introduced in random forests by Breiman. To calculate a variable’s permutation importance, the given variable is randomly permuted in the out-of-sample data (i.e., the observations not selected in the bootstrap random sampling with replacement, called the out-of-bag [OOB] data), and the permuted OOB data are dropped down a tree. OOB prediction error is then calculated. The difference between this and the OOB error without permutation (i.e., from the original tree), averaged over all trees, is the importance of the variable. The larger the permutation importance of a variable, the more predictive the variable, as illustrated in Fig. 7.19 .

Illustration of how randomly permuting a variable leads to different terminal node assignments, which is at the heart of why permutation importance works. Red nodes are tree nodes that split on the target variable v. In the top panel on the left, the bold arrows show the path that a data point x takes as it traverses through the tree to its terminal node assignment “1.” On the right, the v coordinate of x has been randomly permuted, and its new terminal node assignment is now “4.” In the bottom panel is the path for another x value. Its terminal node assignment is “6.” However, when v is permuted, its new terminal assignment becomes “5.” Importantly, notice that in the top panel the terminal node assignment after randomly permuting v is much farther than its original terminal node assignment than in the bottom panel. This shows that the higher v splits in the tree, the more effect permutation has on final terminal node membership, and hence on prediction error.

Other approaches not using prediction error have also been developed for selecting variables; however, these tend to be specifically designed for the algorithm being considered. Recently, attention has been given to developing variable importance that can apply more generally across different types of learning procedures within the framework of model selection. Related to this are methods developed within the framework of model-free feature screening in which explanatory variables are identified without producing an overall prediction model (see Li and colleagues for a discussion of the difference between model selection and variable selection).

A new promising method called variable priority (VarPro) takes a broader approach in the spirit of these latter methods. An interesting aspect of VarPro is that it does not assume linearity or other specific model formulations often used for the conditional means used in model-free feature screening algorithms, while on the other hand, it does construct trees just like in model-selection methods using trees; however, the goal, which is to construct neighborhoods of the covariate space rather than predicting the outcome, is different. Therefore, this method is called model-independent, in the spirit of borrowing the best parts of both model selection and model-free variable selection. Indeed, “variable selection” is a misnomer, because the inner working of VarPro is to progressively exclude (give zero weight to) variables that contribute nothing to predicting outcome, leaving behind those that do. Using rules from externally constructed trees, the VarPro importance statistic for a set of variables equals the difference between the estimator of conditional mean based on a rule and the estimator based on the released rule obtained by removing any constraints on the variables of interest. VarPro readily scales to large data sets requiring only calculating sample averages and can be used in a variety of settings, including regression, classification, and survival.

Calibration.

In logistic regression (see “ Logistic Regression Analysis ” later in this section), a number of diagnostic tools are available. One of the earliest was the decile table, often attributed to Hosmer and Lemeshow but used much earlier by the Framingham investigators and others. By solving the multivariable equation for each patient, patients are ordered with respect to their estimated probability of experiencing an event. They are then stratified in up to 10 groups (thus “decile”), and within each group the estimated probabilities are summed. This sum represents expected events; it is compared with observed events in each decile. The Hosmer-Lemeshow statistic is a general calibration test of the differences between observed and predicted events ( Box 7.19 ).

• BOX 7.19

Calibration and Discrimination

Calibration

Calibration is the process of determining if the results predicted by a model are consistent with the actual data, the closeness (goodness) of fit of a model to data. Hosmer and Lemeshow introduced a method to test the goodness of fit of expected proportion of events as predicted from such regression analyses to observed proportion of events. ^, They proposed a simple way to do this was to “bin” data into deciles (10 groups) of ascending predicted proportion of events and determine within each decile the proportion of actual events observed. A good fitting model would line these up on the diagonal. Deviation from that line was tested by the simple chi-squared goodness of fit test. Despite advances since, this is still a useful tool to explore non-linearities of continuous variables in a model and presence of interactions between variables.

A calibration metric that is useful for both binary models and time-to-event models is the Briar score , ^, which quantifies the differences between predicted probability ( p ) and observed events ( o ):

Briar score = 1 n ∑ i = 1 n ( p i − o i ) 2

.

Note that the better the accuracy, the smaller the value of the Briar score. Walsh and colleagues also provide a comparison of calibration methods.

Discrimination

Discrimination is the ability of a model to stratify data into 2 or more distinct classes. The most familiar tests of discrimination is the C-statistic, or area under the curve (AUC) of a receiver operating characteristic curve (ROC). To understand this, consider the following 2×2 table (sometimes called a confusion matrix):

	Event happened (positive)	Event didn’t happen (negative)
Predicted positive	a True positive (TP)	b False positive (FP)
Predicted negative	c False negative (FN)	d True negative (TN)

From this are derived a number of relationships:

• Sensitivity = TP/(TP + FN), detection of true positives

• Specificity = TN/(TN + FP), detection of true negatives

• Positive predictive value = TP/(TP + FP), for those testing positive, proportion actually positive

• Negative predictive value = TN/(TN + FN), for those testing negative, proportion actually negative

• Precision = TP/(TP + FP), which is identical to positive predictive value

• Recall = TP/(TP + FN), which is identical to sensitivity

ROC is plotted as sensitivity vs. 1-specificity and ROC-AUC is the area under the relationship of sensitivity to 1-specificity.

Trouble begins, however, when events, such as those after cardiac surgery become small and its denominator proportionately large (known as imbalanced data), as is demonstrated in detail under Classification Using Machine Learning. In such a case, false negatives become a large problem, and this is particularly for machine learning for classification, whose predictions will favor the majority class. This suggests a focus on the minority class, which is what recall (sensitivity) does along with precision (positive predictive value). As data become increasingly imbalanced, ROC-AUC increases to approach 1, even though its prediction of the outcome of interest may be poor, because this is overwhelmed by all the true negatives with hardly any false positives (sensitivity). In other words, sensitivity is focused on how well a model detects true positives, and precision is focused on how well the model avoids the false positives. Thus, for imbalanced data, one wants the area under the precision-recall curve (PR-AUC) to be approaching 1.

There are a multitude of tests for discrimination in addition to ROC-AUC and PR-AUC. For time-related events, Harell’s Concordance Index is analogous to ROC-AUC. G-mean is the square root of the product of sensitivity and specificity, which is a measure of poor performance in classifying the minority class, because if there are few false positives, specificity will approach 1 while false negatives are emphasized by the sensitivity. F1 measures the balance between precision and sensitivity and is 2 times the product of sensitivity and precision divided by the sum of sensitivity and precision.

One of the best sources of information on calibration and discrimination and other metrics for goodness of model fit is a series of articles by Frank Harrell and colleagues. ^{, ,} For machine learning, O’Brien and Ishwaran have provided a method to generate accurate probabilities in the face of class imbalance.

Discrimination.

Borrowing from classification theory, which deals with false and true positives and false and true negatives, an analysis of sensitivity and specificity can be performed, varying the cut point from 0 to 1 of what probability is considered predictive of an event occurring. The number of correctly predicted events (true positives) divided by the number of true positives plus false negatives is sensitivity. The number of correctly predicted nonevents (true negatives) divided by the number of true negatives plus false positives is specificity. A graph of 1-specificity on the horizontal axis and sensitivity on the vertical axis is then constructed—the receiver operating characteristic (ROC) curve ( Fig. 7.20 ). The area beneath this curve is a measure of goodness of fit (discrimination, see Box 7.19 ). ^, (Harrell and colleagues called this the c index of concordance. ) It varies from.5 to 1. A concordance index between.8 and.9 is desirable for prediction purposes.

Illustration of receiver operating characteristic (ROC) curves. These are for renal failure after either coronary artery bypass grafting or cardiac valve procedures in 15,844 patients operated on at Cleveland Clinic from 1986 to 2000. Three ROC curves are shown. One is based on preoperative laboratory measurements alone, the second on extensive clinical data alone, and the third combines the two. Diagonal dashed line is the line of random prediction.

Other model diagnostics.

In addition, for all varieties of multivariable models, a number of regression diagnostic procedures are used, including formal testing of goodness of fit, identification of observations that particularly influence the results, and analysis of residuals (the difference between observed and predicted values) in linear regression.

However, as risk of cardiac surgery approaches zero (in terms of mortality and several morbidities), traditional ROC curves as a measure of discrimination become deceptive, yielding a misleading metric of accuracy. This is because the outcomes are highly imbalanced (few events among a large number of patients), and simply predicting no events will result in a high metric of accuracy. There are several metrics that mitigate this problem (see Box 7.19 ). One is the graph of precision versus recall. Precision is another term for sensitivity (true positive/[true positive + true negative]), and recall is another term for positive predictive value (true positive/[true positive + false negative]).

A validation technique, part of calibration, that holds future promise is OOB prediction error assessment. ^, As noted earlier in this section, one bootstrap sample or average does not select about a third of patients. These nonselected patients are known as the OOB sample. A model developed on the two thirds of data can be applied to the OOB sample and prediction error calculated as VIMP. ^,

Presentation.

A multivariable parametric model analysis generates an enormous amount of information, including:

• •

  The structure of the model and estimates of parameters related to that structure

• •

  A list of risk factors identified

• •

  Magnitude of association of each risk factor with outcome as adjusted for all other variables in the model (these multipliers may be expressed either as the parameter estimates themselves—called model coefficients —or as some reformatted relative risk expression (see Box 7.3 )

• •

  Direction of each relation, positive or negative

• •

  Uncertainty of the associations, generally expressed as standard deviations of the coefficients

• •

  A statistical score on which a P value is based

• •

  P values

• •

  A set of numbers indicating quantitative interrelation of all parameter estimates in the model (the variance–covariance matrix)

• •

  Bootstrap reliability of each risk factor identified

There is some controversy about which of these nine sets of numbers should be reported in a manuscript. It may be sufficient for understanding the relations to simply list the risk factors and place in an appendix some of the numeric data. If the model is intended to be used for prediction, including CLs, the entire list must be reported or provided electronically, as was previously done for Society of Thoracic Surgeons National Cardiac Database models.

None of the nine, however, directly addresses the way a final multivariable model is formulated to reveal incremental risk factors (see “ Incremental Risk Factor Concept ” in text that follows). The incremental risk factor concept was developed to facilitate medical interpretation of a multivariable analysis. Any dichotomous risk factor in a multivariable analysis can be complemented to allow it to have a positive sign. This is desirable because we think of variables in the model as risk factors, and usually we consider risk to be increasing (positive value) with increasing value of the risk factor. Generally, continuous and ordinal variables cannot be formulated this way, so we recommend that each of these be accompanied by an indication of the direction of greater risk (younger age, lower ejection fraction, greater functional impairment, higher bilirubin).

Only gold members can continue reading. Log In or Register to continue

Share this:

• X
• Facebook

Related

Related posts:

1. Anatomy and dimensions
2. Cardiac rhythm disturbances
3. Acute traumatic aortic disruption
4. Origin of the right or left pulmonary artery from the ascending aorta

Stay updated, free articles. Join our Telegram channel

Join