Study Subjects

The ultrasound training program and its integration into an accredited internal medicine residency at Scripps Mercy Hospital in San Diego, California, has been in existence for >10 years and has been described in full detail elsewhere. Since 2005, all residents who were accepted in the internal medicine program (approximately 10 residents each year) participated in the 3-year ultrasound curriculum. No resident had previous experience in ultrasound training before residency.

Ultrasound Training and Testing

The CLUE training program consisted of teaching seven core views on two-dimensional ultrasound with nine simplified, evidence-based signs of disease ( Table 1 ) through monthly lectures, Web-based learning, and clinical rotations, including two cardiology consultation and 6 intensive care unit months. Each resident typically received a minimum of 50 hours of didactic or bedside instruction and performed approximately 60 CLUE examinations.

Just before graduation from residency, all residents were required to take a skill and knowledge assessment clinical exercise on CLUE, named the CLUE-CEX. The CLUE-CEX testing procedure, modeled after the “mini-CEX” procedure endorsed by the American Board of Internal Medicine to assess resident performance, required the resident to image a patient in front of a faculty referee and to verbally answer standardized questions regarding each of the seven core views obtained on the patient. The overall CLUE-CEX score, expressed as a percentage of a perfect score, contained separate assessments of (1) image acquisition and quality, scored 0 to 4 points (0 = no image; 1 = gross, off-axis errors; 2 = suboptimal or partial image; 3 = adequate for diagnosis; 4 = optimized image) per view and accounting for 44% of total points, (2) knowledge of diagnostic criteria, scored 0 to 2 points (0 = absent, 1 = partial, 2 = complete) per view and accounting for 28% of total points, and (3) image interpretation, scored 0 to 2 points (0 = inaccurate, 1 = partial credit, 2 = accurate) per view and accounting for 28% of total points. In cases in which the patient had a difficult window, the resident’s acquisition score was normalized to that of the faculty referee’s maximal score attained on that view. Determined over the initial 3 years of the program through faculty observation of resident practice and expert opinion, an 80% correct test score was felt to represent competency in the core concepts of the CLUE curriculum and had parity to other thresholds in resident skill assessments. Therefore, the 80% threshold has been considered a passing score for the CLUE-CEX since 2008. For convenience, the residents performed their tests using either a standard echocardiograph or a portable, laptop-based or hand-carried device that was available at the time, often determined by the location of the patient they were tested upon (e.g., outpatient clinic, echocardiography laboratory, inpatient wards, intensive care unit).

Study Design

Since the inception of CLUE-CEX testing in 2008, all residents ( n = 74) have participated, with 82% successfully “passing” at graduation. Of the 61 residents who had completed the ultrasound training program and passed the CLUE-CEX, and were therefore considered proficient, 32 remained in local practice and were contacted in random order, of whom 30 agreed to take a repeat CLUE-CEX, many on the contingency that it could be performed at their offices or homes. One graduate was on maternity leave, and one could not be located. To attest to the overall competency of the 30 participating physicians, 100% of these graduates had passed upon their initial sitting for internal medicine board certification from the American Board of Internal Medicine.

The first 20 graduates were specifically told not to review any material in preparation and were designated as the NOPREP group. The subsequent 10 physicians, designated as the PREP group, were encouraged to review summary CLUE videos online, which included brief “how to” summary lessons on proper image acquisition, interpretation, and diagnostic criteria. Physicians were advised not to communicate with colleagues about the test and were reassured that their performance scores were anonymous. No incentive was given to any physician in order to motivate good performance.

Physician graduates were privately retested using the same CLUE-CEX format with either a standard echocardiographic machine (iE33; Philips Healthcare, Andover, MA) at the hospital echocardiography laboratory ( n = 20) or, if in their offices or homes ( n = 10), with a portable pocket-sized ultrasound device (Vscan Duoprobe; GE Healthcare, Wauwatosa, WI), on one of two normal volunteers with good acoustic windows. No data were available on the type of equipment the residents were originally tested on. The repeat CLUE-CEX assessments were made by the same faculty member who had performed the original CLUE-CEX, who was blinded to the graduates’ original scores. Physicians were asked to state the last time they performed cardiac ultrasound since graduation and state their current specialties accounting for the majority of their practice. The faculty referee was also allowed to write notes on the graduates’ performance for later review.

Scores from the CLUE-CEX were subcategorized as skill-based, through image acquisition quality subsection scores, and as knowledge-based, through subsection scores of diagnostic criteria, and further stratified by the six separate CLUE sections. The interpretation subsection scores were not separately analyzed, because this category was subject to the bias of imaging a normal volunteer and was somewhat dependent upon the quality of the image acquired by the physician. However, because point-of-care ultrasound requires physician interpretation while imaging, to mitigate the overlap with image acquisition skill scoring, the interpretative skill score had been assessed by having the physician identify the supposed anatomy on whatever image he or she obtained, regardless of its quality. The same original CLUE-CEX data reporting form was used for retesting and has been published.

Statistical Analysis

Overall score was calculated as percentage correct by adding image acquisition and quality, knowledge, and interpretation points for each view divided by the sum of the maximum points possible for each item, multiplied by 100%. Data are reported as mean percentage correct ± SD. After testing all the graduates, the original CLUE-CEX results for each were compared with the graduate’s current results and plotted against the time given. Retention was defined as the ratio of the recent CLUE-CEX score to the original CLUE-CEX total or subsection score and was expressed as a percentage. A 3 × 3 contingency table of image acquisition scores versus knowledge scores in the NOPREP ( n = 20) group was assessed by χ ² analysis. The difference in the CLUE-CEX performance between the NOPREP and PREP groups was compared using Fisher’s exact test for categorical data and Student’s t test for continuous data. A P value ≤ .05 was considered to indicate statistical significance. Statistical analyses were performed using SAS version 9.2 (SAS Institute, Cary, NC) by a dedicated statistician.