Study Institutions

CPMC is a community-based Sutter Health-affiliated hospital with noninvasive pediatric cardiology services including inpatient and outpatient echocardiograms. Cardiac sonographers who perform pediatric echocardiograms at CPMC are credentialed as Registered Diagnostic Cardiac Sonographers with an adult and/or pediatric specialty. The pediatric echocardiography laboratory was initially accredited through IAC in April 2013. In 2013, the volume of all pediatric echocardiograms at CPMC was 2,016.

LPCH at Stanford University School of Medicine provides comprehensive pediatric cardiology subspecialty services, including cardiothoracic surgery, and is one of the major pediatric cardiac referral centers in the United States. Cardiac sonographers (credentialed as Registered Diagnostic Cardiac Sonographers pediatric) and physician trainees (assisted by sonographers) perform echocardiograms at LPCH. It has been accredited through IAC since 2004. In 2013, the volume of all inpatient and outpatient pediatric echocardiograms at LPCH was 10,604.

Study approval was obtained from the institutional review boards of Sutter Health and Stanford University.

Study Population

Preaccreditation echocardiograms from CPMC were included from the time period October 2009 to April 2012. To account for potential bias during the CPMC IAC application preparation and consideration time period, echocardiograms from May 2012 to April 2013 were excluded. Postaccreditation echocardiograms were included from May 2013 to April 2015. Consecutive CPMC patients with congenital heart disease who required cardiac intervention (catheterization or surgery) were identified prospectively and had diagnoses confirmed by cardiac catheterization, computed tomography, magnetic resonance imaging, and/or cardiac surgery during the CPMC pre- and postaccreditation time periods. The CPMC cases were matched with the LPCH cases for age, complexity, and time period. Echocardiograms from LPCH during equivalent time periods were selected, all of which represented postaccreditation studies ( Figure 1 ). The most complete preintervention echocardiogram was selected. Exclusion criteria included normal echocardiograms, echocardiograms performed between May 2012 and April 2013, and a few extremely limited echocardiograms due to agitated/unstable patients. Categorization of cases for complexity was performed based on this study population on a scale of 1 to 3 (1 = minor complexity, 2 = moderate complexity, and 3 = severe complexity), and examples of cardiac diagnoses are delineated in Table 1 . Category 1 included minor complexity defects generally requiring a single catheterization or surgery, Category 2 included moderate complexity defects with two ventricles requiring surgical repair in infancy and possible subsequent surgery, and category 3 included severe complexity defects with one ventricle and/or complex anatomy requiring multiple catheterizations and surgeries.

Study comparisons. Time frames for study selection are era 1, October 2009 to April 2012, and era 2, May 2013 to April 2015. Post-IAC, echocardiograms after accreditation with the IAC; Pre-IAC, echocardiograms before accreditation with the IAC.

Data Collection

The demographic data obtained from the echocardiogram reports and medical records included age, weight, gender, cardiac diagnoses, history of prior intervention, performing sonographer/physician, interpreting physician, study location, date of study, time of study, and date of intervention. Comments regarding agitation, poor acoustic windows, and sedation were also recorded.

Four QMs were used to evaluate echocardiograms for (1) diagnostic accuracy, (2) image quality, (3) study comprehensiveness, and (4) report completeness. The diagnostic accuracy, image quality, and study comprehensiveness metrics were developed by the ACC Adult Congenital Pediatric Cardiology Quality Metrics Working Group initiative. Permission was obtained from the ACC administration to trial the metrics in this study. Report completeness was an IAC tool for a MOC Part 4 quality improvement activity approved by the American Board of Pediatrics.

Diagnostic Accuracy

The diagnostic accuracy metric is based on a taxonomy developed by Benavidez et al. for error categorization and risk factor identification. For this study, the most complete preintervention echocardiogram was selected. The echocardiogram images and report were compared with findings in the medical record from the date of the echocardiogram until 2 weeks after intervention (e.g., inpatient/outpatient progress notes, subsequent echocardiograms, computed tomography/magnetic resonance imaging reports, cardiac catheterization reports, operative notes, and hospital discharge summaries). Data were collected regarding discrepancies in diagnoses between the preintervention echocardiogram report and the subsequent medical record, method of discovery of error, time from the echo to discovery of error, and anatomic segment involved in the error. Each error was discussed among the study investigators to arrive at a consensus regarding categorization of the error into clinical impact (minor/moderate/severe) and preventability (preventable/possibly preventable/not preventable) and to identify contributors to error, which were tabulated. As per the metric, minor clinical impact was defined as no change in patient management, moderate was a change in patient management without adverse event, and severe was a change in patient management with an adverse event. Preventable errors could have been detected by echocardiogram (i.e., visualized in the images obtained but not reported), possibly preventable could have been detected by echocardiogram with a reasonably different imaging technique (i.e., use of color Doppler), and not preventable errors could not have been detected by echocardiogram.

Image Quality

The image QM requires detailed review of echocardiogram images. Two echocardiographers, one from CPMC (S.K.B.) and one from LPCH (T.A.T.), reviewed both CPMC and LPCH echocardiograms for image quality. To achieve uniformity in grading echocardiograms using the image QM, the raters reviewed several trial echocardiograms not included in the study and discussed the grading until consensus prior to study initiation. The study echocardiograms were loaded to the same server, and the reviewers were blinded to the report data but not to study location due to obvious differences in imaging protocols.

The image QM consists of an evaluation of image orientation, two-dimensional (2D) imaging, color flow imaging, and spectral Doppler display. Image orientation assesses whether standard pediatric transthoracic echocardiogram views were obtained with two to five imaging requirements listed per view. If all required elements were obtained for any given view, a score of 1 was given. Partial credit was given if some elements in any given view were obtained, and a score of 0 was given when the view was not obtained. The quality of 2D imaging, color flow imaging, and spectral Doppler components have three elements per imaging technique and were scaled on a continuum of 1 = agree, 0.5 = somewhat agree, and 0 = disagree, as these remain somewhat subjective assessments. Elements for (1) 2D imaging include brightness level, balanced penetration to resolution, presentation of the region of interest; elements for (2) color flow imaging include frame rate, gain level, and Nyquist limit settings; and elements for (3) spectral Doppler include choice of pulsed wave or continuous wave Doppler, gain level, and Doppler profile display.

Study Comprehensiveness

The study comprehensiveness metric also requires in-depth review of echocardiogram images. This was initially performed on echocardiograms not included in the study to establish internal standards and scoring methods between the two echocardiographers (S.K.B. and T.A.T.). The study comprehensiveness metric encompasses the imaging of segmental cardiac anatomy: abdominal situs, cardiac position, systemic and pulmonary venous return, coronary sinus, atria and ventricular septae, atrioventricular valves, semilunar valves, coronary arteries, right ventricle and outflow tract, left ventricle and outflow tract, aortic root, branch pulmonary arteries, aortic arch, abdominal aorta, and effusion. Relevant imaging components per cardiac structure are included such as 2D imaging, color, and spectral Doppler, with multiple structures requiring imaging in more than one view. If all imaging components per structure were performed in the requested number of views, a score of 1 was given. Partial credit was given if some but not all imaging components and/or the requested number of views were obtained, and a score of 0 was given when the structure was not imaged.

Report Completeness

The reports from all study echocardiograms were reviewed for completeness using an IAC form approved for a Part 4 MOC activity. This form includes (1) demographic data: date of study, laboratory identifier, patient identifiers and characteristics (height, weight, gender, blood pressure), study indication, ordering physician, and performing sonographer; (2) 2D or M-mode measurements: left ventricle, septal, posterior wall, and aortic root dimensions; (3) Doppler evaluation: valve regurgitation, peak and mean gradients in the presence of stenosis, and right ventricular systolic pressure in the presence of tricuspid regurgitation; and (4) report components: standardization with other reports, summary inclusion, typewritten report, and interpreting physician name and signature. A score of 1 was given for each element included in the report. Reporting regurgitation (or lack thereof) was required for at least one valve to obtain one point for the regurgitation element. We reviewed the dichotomous scoring of the report completeness metric with an IAC representative and performed analyses consistent with the IAC.

Final Scoring of QMs

Discrepancies in diagnoses that were determined to be errors were qualitatively recorded and categorized. For diagnostic errors, the diagnostic error rate (as recommended by the diagnostic accuracy metric) was calculated as the number of errors of moderate clinical impact or greater that were possibly preventable or preventable over the total number of echocardiograms studied. For image quality, study comprehensiveness, and report completeness, if there was a component of the metric that was not applicable for the specific case, a point was deducted from the total number of points possible. Final scoring was determined as a percentage of total number of points obtained divided by the total number of possible points. Since this is the first application of these metrics to assess quality, we compared partial credit with dichotomous scoring (0 vs 1) for image quality and study comprehensiveness, both of which have multiple elements listed for each requirement.

Data Analysis

Patient demographics were characterized for each group of echocardiograms during the two designated eras: CPMC pre-IAC accreditation (era 1) and CPMC post-IAC accreditation (era 2) at both sites (CPMC and LPCH). The four comparisons performed in statistical analyses of the metrics are depicted in Figure 1 : (A) CPMC pre-IAC accreditation versus CPMC post-IAC accreditation, (B) CPMC pre-IAC accreditation versus LPCH post-IAC accreditation during the equivalent era 1, (C) CPMC post-IAC accreditation versus LPCH post-IAC accreditation during the equivalent era 2, and (D) LPCH post-IAC accreditation in era 1 versus LPCH post-IAC accreditation in era 2 (performed to account for era bias).

Diagnostic errors (the number of echocardiograms with errors divided by total number of echocardiograms) were calculated for each group and compared between groups with χ ² analyses to assess the association between outcome and predictors. Clinical impact and preventability were compared between groups using the Cochran-Armitage test. The equality of proportions was tested to compare diagnostic error rates (the number of errors of moderate clinical impact or greater that were possibly preventable or preventable over the total number of echocardiograms studied). Image quality and study comprehensiveness had scores using both partial and dichotomous scoring methods, from two raters. The scores were averaged (each scoring method was independently analyzed) and used in the comparisons described in Figure 1 . Pearson correlation was performed to evaluate scores from the two raters. Results from the image quality, study comprehensiveness, and report completeness metrics did not appear normally distributed so we used the Mann-Whitney-Wilcoxon nonparametric test to assess whether two independent samples of observations were drawn from the same or identical distributions.

Study Institutions

CPMC is a community-based Sutter Health-affiliated hospital with noninvasive pediatric cardiology services including inpatient and outpatient echocardiograms. Cardiac sonographers who perform pediatric echocardiograms at CPMC are credentialed as Registered Diagnostic Cardiac Sonographers with an adult and/or pediatric specialty. The pediatric echocardiography laboratory was initially accredited through IAC in April 2013. In 2013, the volume of all pediatric echocardiograms at CPMC was 2,016.

LPCH at Stanford University School of Medicine provides comprehensive pediatric cardiology subspecialty services, including cardiothoracic surgery, and is one of the major pediatric cardiac referral centers in the United States. Cardiac sonographers (credentialed as Registered Diagnostic Cardiac Sonographers pediatric) and physician trainees (assisted by sonographers) perform echocardiograms at LPCH. It has been accredited through IAC since 2004. In 2013, the volume of all inpatient and outpatient pediatric echocardiograms at LPCH was 10,604.

Study approval was obtained from the institutional review boards of Sutter Health and Stanford University.

Study Population

Preaccreditation echocardiograms from CPMC were included from the time period October 2009 to April 2012. To account for potential bias during the CPMC IAC application preparation and consideration time period, echocardiograms from May 2012 to April 2013 were excluded. Postaccreditation echocardiograms were included from May 2013 to April 2015. Consecutive CPMC patients with congenital heart disease who required cardiac intervention (catheterization or surgery) were identified prospectively and had diagnoses confirmed by cardiac catheterization, computed tomography, magnetic resonance imaging, and/or cardiac surgery during the CPMC pre- and postaccreditation time periods. The CPMC cases were matched with the LPCH cases for age, complexity, and time period. Echocardiograms from LPCH during equivalent time periods were selected, all of which represented postaccreditation studies ( Figure 1 ). The most complete preintervention echocardiogram was selected. Exclusion criteria included normal echocardiograms, echocardiograms performed between May 2012 and April 2013, and a few extremely limited echocardiograms due to agitated/unstable patients. Categorization of cases for complexity was performed based on this study population on a scale of 1 to 3 (1 = minor complexity, 2 = moderate complexity, and 3 = severe complexity), and examples of cardiac diagnoses are delineated in Table 1 . Category 1 included minor complexity defects generally requiring a single catheterization or surgery, Category 2 included moderate complexity defects with two ventricles requiring surgical repair in infancy and possible subsequent surgery, and category 3 included severe complexity defects with one ventricle and/or complex anatomy requiring multiple catheterizations and surgeries.

Study comparisons. Time frames for study selection are era 1, October 2009 to April 2012, and era 2, May 2013 to April 2015. Post-IAC, echocardiograms after accreditation with the IAC; Pre-IAC, echocardiograms before accreditation with the IAC.

Data Collection

The demographic data obtained from the echocardiogram reports and medical records included age, weight, gender, cardiac diagnoses, history of prior intervention, performing sonographer/physician, interpreting physician, study location, date of study, time of study, and date of intervention. Comments regarding agitation, poor acoustic windows, and sedation were also recorded.

Four QMs were used to evaluate echocardiograms for (1) diagnostic accuracy, (2) image quality, (3) study comprehensiveness, and (4) report completeness. The diagnostic accuracy, image quality, and study comprehensiveness metrics were developed by the ACC Adult Congenital Pediatric Cardiology Quality Metrics Working Group initiative. Permission was obtained from the ACC administration to trial the metrics in this study. Report completeness was an IAC tool for a MOC Part 4 quality improvement activity approved by the American Board of Pediatrics.

Diagnostic Accuracy

The diagnostic accuracy metric is based on a taxonomy developed by Benavidez et al. for error categorization and risk factor identification. For this study, the most complete preintervention echocardiogram was selected. The echocardiogram images and report were compared with findings in the medical record from the date of the echocardiogram until 2 weeks after intervention (e.g., inpatient/outpatient progress notes, subsequent echocardiograms, computed tomography/magnetic resonance imaging reports, cardiac catheterization reports, operative notes, and hospital discharge summaries). Data were collected regarding discrepancies in diagnoses between the preintervention echocardiogram report and the subsequent medical record, method of discovery of error, time from the echo to discovery of error, and anatomic segment involved in the error. Each error was discussed among the study investigators to arrive at a consensus regarding categorization of the error into clinical impact (minor/moderate/severe) and preventability (preventable/possibly preventable/not preventable) and to identify contributors to error, which were tabulated. As per the metric, minor clinical impact was defined as no change in patient management, moderate was a change in patient management without adverse event, and severe was a change in patient management with an adverse event. Preventable errors could have been detected by echocardiogram (i.e., visualized in the images obtained but not reported), possibly preventable could have been detected by echocardiogram with a reasonably different imaging technique (i.e., use of color Doppler), and not preventable errors could not have been detected by echocardiogram.

Image Quality

The image QM requires detailed review of echocardiogram images. Two echocardiographers, one from CPMC (S.K.B.) and one from LPCH (T.A.T.), reviewed both CPMC and LPCH echocardiograms for image quality. To achieve uniformity in grading echocardiograms using the image QM, the raters reviewed several trial echocardiograms not included in the study and discussed the grading until consensus prior to study initiation. The study echocardiograms were loaded to the same server, and the reviewers were blinded to the report data but not to study location due to obvious differences in imaging protocols.

The image QM consists of an evaluation of image orientation, two-dimensional (2D) imaging, color flow imaging, and spectral Doppler display. Image orientation assesses whether standard pediatric transthoracic echocardiogram views were obtained with two to five imaging requirements listed per view. If all required elements were obtained for any given view, a score of 1 was given. Partial credit was given if some elements in any given view were obtained, and a score of 0 was given when the view was not obtained. The quality of 2D imaging, color flow imaging, and spectral Doppler components have three elements per imaging technique and were scaled on a continuum of 1 = agree, 0.5 = somewhat agree, and 0 = disagree, as these remain somewhat subjective assessments. Elements for (1) 2D imaging include brightness level, balanced penetration to resolution, presentation of the region of interest; elements for (2) color flow imaging include frame rate, gain level, and Nyquist limit settings; and elements for (3) spectral Doppler include choice of pulsed wave or continuous wave Doppler, gain level, and Doppler profile display.

Study Comprehensiveness

The study comprehensiveness metric also requires in-depth review of echocardiogram images. This was initially performed on echocardiograms not included in the study to establish internal standards and scoring methods between the two echocardiographers (S.K.B. and T.A.T.). The study comprehensiveness metric encompasses the imaging of segmental cardiac anatomy: abdominal situs, cardiac position, systemic and pulmonary venous return, coronary sinus, atria and ventricular septae, atrioventricular valves, semilunar valves, coronary arteries, right ventricle and outflow tract, left ventricle and outflow tract, aortic root, branch pulmonary arteries, aortic arch, abdominal aorta, and effusion. Relevant imaging components per cardiac structure are included such as 2D imaging, color, and spectral Doppler, with multiple structures requiring imaging in more than one view. If all imaging components per structure were performed in the requested number of views, a score of 1 was given. Partial credit was given if some but not all imaging components and/or the requested number of views were obtained, and a score of 0 was given when the structure was not imaged.

Report Completeness

The reports from all study echocardiograms were reviewed for completeness using an IAC form approved for a Part 4 MOC activity. This form includes (1) demographic data: date of study, laboratory identifier, patient identifiers and characteristics (height, weight, gender, blood pressure), study indication, ordering physician, and performing sonographer; (2) 2D or M-mode measurements: left ventricle, septal, posterior wall, and aortic root dimensions; (3) Doppler evaluation: valve regurgitation, peak and mean gradients in the presence of stenosis, and right ventricular systolic pressure in the presence of tricuspid regurgitation; and (4) report components: standardization with other reports, summary inclusion, typewritten report, and interpreting physician name and signature. A score of 1 was given for each element included in the report. Reporting regurgitation (or lack thereof) was required for at least one valve to obtain one point for the regurgitation element. We reviewed the dichotomous scoring of the report completeness metric with an IAC representative and performed analyses consistent with the IAC.

Final Scoring of QMs

Discrepancies in diagnoses that were determined to be errors were qualitatively recorded and categorized. For diagnostic errors, the diagnostic error rate (as recommended by the diagnostic accuracy metric) was calculated as the number of errors of moderate clinical impact or greater that were possibly preventable or preventable over the total number of echocardiograms studied. For image quality, study comprehensiveness, and report completeness, if there was a component of the metric that was not applicable for the specific case, a point was deducted from the total number of points possible. Final scoring was determined as a percentage of total number of points obtained divided by the total number of possible points. Since this is the first application of these metrics to assess quality, we compared partial credit with dichotomous scoring (0 vs 1) for image quality and study comprehensiveness, both of which have multiple elements listed for each requirement.

Data Analysis

Patient demographics were characterized for each group of echocardiograms during the two designated eras: CPMC pre-IAC accreditation (era 1) and CPMC post-IAC accreditation (era 2) at both sites (CPMC and LPCH). The four comparisons performed in statistical analyses of the metrics are depicted in Figure 1 : (A) CPMC pre-IAC accreditation versus CPMC post-IAC accreditation, (B) CPMC pre-IAC accreditation versus LPCH post-IAC accreditation during the equivalent era 1, (C) CPMC post-IAC accreditation versus LPCH post-IAC accreditation during the equivalent era 2, and (D) LPCH post-IAC accreditation in era 1 versus LPCH post-IAC accreditation in era 2 (performed to account for era bias).

Diagnostic errors (the number of echocardiograms with errors divided by total number of echocardiograms) were calculated for each group and compared between groups with χ ² analyses to assess the association between outcome and predictors. Clinical impact and preventability were compared between groups using the Cochran-Armitage test. The equality of proportions was tested to compare diagnostic error rates (the number of errors of moderate clinical impact or greater that were possibly preventable or preventable over the total number of echocardiograms studied). Image quality and study comprehensiveness had scores using both partial and dichotomous scoring methods, from two raters. The scores were averaged (each scoring method was independently analyzed) and used in the comparisons described in Figure 1 . Pearson correlation was performed to evaluate scores from the two raters. Results from the image quality, study comprehensiveness, and report completeness metrics did not appear normally distributed so we used the Mann-Whitney-Wilcoxon nonparametric test to assess whether two independent samples of observations were drawn from the same or identical distributions.