Penn and DCRI ECL Group Reads

To align data collection elements, the Penn and DCRI ECLs reviewed two-dimensional (2D) and Doppler echocardiographic parameters of cardiac size and systolic and diastolic function relevant to clinical cardiovascular outcomes in patients with cancer. A harmonization process between ECLs ensued. Sonographers and principal investigators at both ECLs conducted serial calls and multiple Web-conference group reads from October 2013 to March 2014 to share standard operating procedures, review sample echocardiograms, and align ECL perspectives on image quality, border selection, and tracing conventions. Group readings illustrated differences between ECLs on tracing conventions for certain parameters. These included nonconsensus on the angle of the minor axis for LV internal dimensions (e.g., parallel to mitral valve plane vs perpendicular to the LV long axis) and LV endocardial border definitions (e.g., depth of exclusion of trabeculae) during measurements of LV internal dimensions ( Figure 1 ) and volumes ( Figure 2 ), respectively. Harmonization efforts were aimed at achieving consensus on the application of measurement conventions outlined in national guidelines recommendations and culminated in the development of consensus reading instructions (please see the Online Appendix , available at www.onlinejase.com ) as well as a common comprehensive case report form across ECLs.

LV internal dimension tracing conventions by ECL.

LV endocardial border tracing conventions by ECL.

Echocardiographic Acquisition and Creation of Analysis Repository

After developing consensus reading instructions, each ECL contributed echocardiograms for reproducibility analyses. A total of 25 patient echocardiograms were selected from transthoracic echocardiograms previously acquired at both institutions from patients who had completed treatment with potentially cardiotoxic anticancer agents (i.e., doxorubicin with or without trastuzumab) for breast cancer. More detailed clinical data were not made available, as each ECL was blinded to patient characteristics. Selected echocardiograms were required to have visible LV endocardium unobscured by undergaining or artifact and no significant apical foreshortening in 2D acquisitions. Echocardiograms were obtained by dedicated sonographer teams in the Intersocietal Accreditation Commission clinical laboratories at both institutions. All images were acquired using Vivid 7 or E9 machines (GE Healthcare, Milwaukee, WI) at 60 to 90 frames/sec and digitally archived at the acquisition frame rate. Digital echocardiographic images were deidentified and transferred in standard Digital Imaging and Communications in Medicine format to TomTec (TomTec Imaging Systems, Unterschleissheim, Germany) and DigiView (Digisonics, Houston, TX) analysis workstations at the Penn and DCRI ECLs, respectively. Both ECLs used TomTec 2D Cardiac Performance Analysis version 1.1 for strain analysis.

Measurement of Echocardiography Parameters

After image transfer, measurements of echocardiography parameters were assigned to two readers at each ECL ( n = 4 total readers). Readers included three highly experienced research sonographers and a cardiologist with level III certification in echocardiography. Each reader independently analyzed two uniquely identified copies of the 25 patient echocardiograms and recorded 50 measurement results per analyzed parameter. Each result was treated independently (i.e., no averaging within or between readers).

At laboratory A, LV volumes and strain were measured by a single reader; Doppler parameters (i.e., velocities and timing intervals) were measured by a separate reader, according to reader expertise and existent laboratory practices. At laboratory B, each reader measured every parameter. The measurement results generated per echocardiogram by the readers in each ECL are depicted in more detail in Figure 3 A.

ECL assignment of echocardiography measurements and comparisons of results. (A) For each echocardiogram, there were two results per reader for every measured parameter. At laboratory A, LV volumes and strain were measured by a single reader; Doppler parameters were measured by a separate reader. At laboratory B, each reader measured every parameter. (B) Pairwise comparisons of measurement results between the two readers at laboratory B. For each measured parameter per echocardiogram, there were four pairwise comparisons of two measurement results per reader between the two readers at laboratory B. (C) Pairwise comparisons of measurement results between the readers at laboratories A and B. For each measured parameter per echocardiogram, there were eight pairwise comparisons of two measurement results per reader between one reader at laboratory A and two readers at laboratory B. 4CH , Apical four-chamber; LVEDV , LV end-diastolic volume; LVESV , LV end-systolic volume; LVOT , LV outflow tract; MV , mitral valve.

Intra- and Interlaboratory Reproducibility Testing

Intra- and interlaboratory reproducibility was evaluated by pairwise comparisons of all possible measurement differences (interpretative variability) among all readers for selected echocardiographic parameters, as described below. For each of the 25 patient echocardiograms, there were four pairwise comparisons of measurement results between the two readers at laboratory B ( Figure 3 B) and eight pairwise comparisons of measurement results between the reader at laboratory A and two readers at laboratory B ( Figure 3 C) on any selected parameter.

The “acceptable difference,” or limit of measurement variability, for each parameter served as a benchmark for reproducibility. The intra- and interreader acceptable differences were defined prospectively on the basis of literature review of studies with reproducibility data for LV volumes and LVEF, mitral inflow velocities, global longitudinal strain (GLS), and R-wave flow onset and flow end times. All acceptable differences are expressed in absolute terms ( Table 1 ).

After readers completed the echocardiographic interpretations, result reports were generated, and intra- and interreader pairwise comparisons were displayed in tables and illustrated in dot-plot graphs. Paired measurements with differences exceeding the prespecified acceptable differences (outliers) were reviewed in a group read. If reproducibility between readers was not acceptable for any echocardiographic parameter(s), a process of revisiting image analysis instructions, retraining and retesting ensued.

Retraining and Retesting

Retraining involved Web conference–based sessions among all readers. Review of individual statistical results guided the retraining process by illustrating the extent and direction of outlier pairs. Open discussion among readers helped identify the source(s) of interpretation and/or measurement error(s). Illustrative case examples and the consensus reading instructions were discussed and revised to promote uniformity of interpretation and help eliminate individual idiosyncrasies. After retraining, all readers reinterpreted the 25 echocardiograms twice for selected parameters with unacceptable reproducibility on initial testing, and reproducibility was reevaluated.

Statistical Analysis

Specific standards or a universally preferred index does not exist for assessing intrareader and interreader reproducibility in ECLs; therefore, multiple statistical approaches for assessing reproducibility were considered. The coverage probability (CP) method was selected given its desirable characteristics as an agreement index in the ECL setting, including its computational simplicity, rapid identification of group and individual reader variability and specific disagreements for review and retraining, and broad applicability to patient populations and continuous and categorical variables. CP is the probability that the difference between any two measurements of a parameter on the same echocardiogram are within a prespecified acceptable difference. Specifically, all possible interreader comparisons were examined to determine whether the measurement difference between paired readers was within the prespecified acceptable difference. The nonparametric approach was used to obtain the estimate of CP. The estimate of CP is the proportion of the number of pairwise interreader comparisons within the acceptable difference divided by the number of all possible pairwise comparisons. The higher the CP, the better the reproducibility. Perfect reproducibility corresponded to 100% CP (i.e., CP = 1.00), indicating that all measurements were within the prespecified acceptable difference for that parameter.

The prespecified acceptable differences and the cut point for the CP determine the standard for acceptable reproducibility. In this study, the cut point for the estimated CP was set at 0.80 on the basis of previous studies assessing reproducibility in an ECL setting. Reproducibility was considered acceptable if ≥80% of all possible pairwise comparisons were within the prespecified acceptable difference for each parameter. Additionally, all pairwise comparisons (100% or CP = 1.00) were required to be within twice the acceptable difference. To easily visualize the reproducibility data and rapidly identify outlier pairs, the results of the CP analysis were displayed graphically for continuous parameters. Statistical analyses were performed using SAS version 9.2 (SAS Institute, Cary, NC).