Background
Facilitated reporting using a discrete set of finding codes (FCs) is a common method of generating echocardiographic reports.
Methods
The investigators developed a tool that allows echocardiographic reports to be evaluated in real time for errors, omissions, and inconsistencies on the basis of a defined group of “rules” applied to the FCs present in the report. At the time of report finalization, conflicts were displayed for the interpreting physicians, and their responses to each rule conflict were logged.
Results
Over the course of 1 year, 7,986 transthoracic echocardiographic reports were analyzed prospectively during study interpretation. Overall, 30 ± 4.7 FCs were used to generate finalized reports. An average of 2.4 ± 2.0 conflicts were detected per finalized study. Eighty-three percent of studies had at least one conflict identified. There was no significant correlation between physician experience and conflict rates, but time of day (earlier) and rate at which studies were being finalized (faster) were both correlated with increased conflict rate. Overall, physicians ignored identified conflicts 52% of the time and altered their readings to eliminate the conflicts 48% of the time. Overall, at least one change was made in 54% of all finalized studies. There was a small but significant trend for physicians to produce more conflicts over time as the tool was used.
Conclusions
This study demonstrates that facilitated reporting of echocardiographic studies, using a discrete set of FCs, allows the generation of rules that can be used to identify discrepancies in echocardiographic reports before finalization. Conflicts are common in clinical practice, and the identification of these conflicts in real time allowed readers to review their interpretations and frequently resulted in alterations to echocardiographic reports.
In facilitated reporting, a reader selects pertinent statements from delimited drop-down lists of potential finding codes (FCs) to document the interpretation of an imaging study. These FCs are typical short summary statements that denote specific echocardiographic findings and, when selected, populate the clinical report with a complete interpretive sentence. This report generation technique provides consistency among readers, improves report organization, and speeds the turnaround of finalized reports. Although this method avoids the errors inherent in the dictation and transcription process, facilitated reporting is not error free, as physicians can still select FCs that are in conflict with one another. We developed a tool that allows echocardiographic reports to be evaluated for errors, omissions, and inconsistencies on the basis of a defined set of “rules” that are applied to the FCs present in the report. Using this tool, we showed, in a previous retrospective analysis, that errors and inconsistencies are not infrequent in finalized reports. We sought to test whether implementation of this tool in a busy clinical practice could identify errors and inconsistencies in echocardiographic reports at the time of report finalization and how interpreting physicians would interact with the tool notifications.
Methods
A set of “rules” was generated on the basis of the unique FC data set used at a single center. The rule creation tool allowed multiple FC conflicts to be generated with a single rule. For example, “the following FC are mutually exclusive,” if it contained five FCs, would represent 10 individual potential conflicts. Rules were created to detect (1) FCs that should never appear together (e.g., “MV structurally normal” with “MV prolapse,” “no TR” with “mild TR”); (2) FCs that, although possible, would typically not appear together (e.g., “LA normal size” with “moderate diastolic dysfunction,” “LV normal size” with “moderate-severe AR,” “severe AS” with “normal LV mass”); (3) FCs that would have been expected to be present (e.g., “severe diastolic dysfunction” would imply that “LV systolic dysfunction” or “LVH” should be present); and (4) inconsistencies between measurements or calculations and FC (e.g., “LA volume measurement equal to 24 mL/m 2 ” with “LA is moderately enlarged”). A more detailed inventory of the concepts used to generate rules is listed in the Appendix (available at www.onlinejase.com ). Rules were identified as mandatory resolution (MR), meaning that the report could not be finalized without correction of the conflict, or as suggested resolution (SR), in which case the rule conflict could be addressed or ignored at the discretion of the reader. Free-text statements manually entered by the reader, which complement the information conveyed by the FCs, were not evaluated by the tool.
The rule set was run against the FCs present in clinical echocardiographic reports at the time of report finalization, and conflicts were displayed for the interpreting physician. Along with identification of the apparent conflict, the tool suggested possible corrections to the reader. Physicians’ responses to each rule conflict were logged. Responses to MR rules included (1) adding or deleting an FC suggested by the tool and (2) returning to the report to perform manual edits to the report. The logged responses for non-MR rules (SR rules) included (1) adding or deleting an FC suggested by the tool, (2) returning to the report to perform manual edits, and (3) ignoring the rule conflict. After each action or report change, the echocardiographic report was immediately reanalyzed by the tool. This allowed multiple rule conflicts to be corrected by a single change in FC, as well as identification of new rule conflicts introduced by the addition or deletion of a FC.
The five participating physicians consented to participate and were level II and III trained echocardiographers with an average duration of time since fellowship completion of 13 years (range, 7–26 years) and had interpreted an average of 1,706 adult transthoracic studies in the prior year (range, 568–3,393). Physicians were scheduled to interpret clinically ordered studies on a rotating schedule that was not altered for the study. The single reader who created the rule set and had familiarity with all rules was not included in the study. All studies were analyzed and reported using a commercially available program (Xcelera; Philips Medical Systems, Andover, MA) with which all readers had ≥5 years’ experience. Physicians were instructed to interpret studies according to their usual individual styles and informed that during the study period, upon report finalization, an interface would appear notifying them of potential errors, inconsistencies, and omissions from their reports. Data were collected prospectively over a 1-year period. Readers were surveyed after several months of use to assess their perceived impact of the tool on their echocardiographic reading experience.
The numbers of rule conflicts (MR and SR) were tabulated for each study interpreted, and the average rate of conflicts per study was calculated. Interphysician differences were evaluated with analysis of variance. Physician error rates were correlated against reader experience (years since level II American Society of Echocardiography certification). Conflict rates were tabulated according to time of day, as well as the “speed” with which reading was performed, as measured by reports finalized per hour. To evaluate whether there was a learning effect, the average number of conflicts per report was determined on a weekly average and tracked over the duration of the study.
Results
Three hundred fifty MR and 230 SR rules were created. These rules contained 3,258 MR and 1,157 SR individual FC conflicts. Of the individual conflicts, 360 MR and 289 SR conflicts contained measurements. Over the entire course of the study, 1,149 unique FC conflicts were detected on at least one occasion. Eighty-four percent of the MR and 48% of the SR conflicts were never detected in finalized echocardiographic reports.
Over the course of 15 months (June 1, 2011, to August 31, 2013), 7,986 transthoracic echocardiographic reports were analyzed at the time of report finalization. Overall, the readers selected 30 ± 4.7 FCs to generate finalized reports. There were 19,166 conflicts detected, composed of 1149 unique rule conflicts. There was an average of 2.4 ± 2.0 conflicts (range, 0–17) detected per finalized study ( Table 1 ). On average, there were 1.7 ± 1.6 SR and 0.7 ± 0.9 MR conflicts per study. Eighty-three percent of studies had at least one conflict identified, and 17% were finalized without any notification from the tool. There were significant differences ( P < .001) in the individual physicians’ total rates, which ranged from 0.8 ± 1.2 to 2.8 ± 2.1 per study ( Table 2 ). The rate of MR and SR conflicts also varied by physician ( P < .001).
| Number of conflicts | Number of studies | Percentage of studies |
|---|---|---|
| 0 | 1,382 | 17.3 |
| 1 | 1,687 | 21.2 |
| 2 | 1,783 | 22.4 |
| 3 | 1,265 | 15.9 |
| 4 | 844 | 10.6 |
| 5 | 457 | 5.7 |
| 6 | 259 | 3.3 |
| 7 | 143 | 1.8 |
| 8 | 74 | 0.9 |
| 9 | 43 | 0.5 |
| 10 | 21 | 0.3 |
| 11 | 14 | 0.2 |
| 12 | 4 | 0.1 |
| 13 | 5 | 0.1 |
| 14 | 2 | 0.03 |
| 15 | 2 | 0.03 |
| 16 | 0 | 0.00 |
| 17 | 1 | 0.01 |
| Total | 7,986 | 100 |
| Reader | Total | MR | SR |
|---|---|---|---|
| 1 | 2.4 ± 1.9 | 0.4 ± 0.7 | 1.9 ± 1.7 |
| 2 | 2.8 ± 2.0 | 1.0 ± 1.0 | 1.7 ± 1.5 |
| 3 | 1.3 ± 1.5 | 0.4 ± 0.7 | 0.9 ± 1.3 |
| 4 | 1.4 ± 1.7 | 0.3 ± 0.6 | 1.0 ± 1.4 |
| 5 | 0.82 ± 1.2 | 0.2 ± 0.5 | 0.6 ± 1.0 |
There was no significant correlation between physician experience (years since level II certification) and conflict rates. There was a small variation in the length of echocardiographic reports by physician (as measured by number of FCs) from 28 ± 5 to 32 ± 5 ( P < .001). However, the variation in physician conflict rate was not explained by the small differences in average number of FCs per study by reader. When taken as a group, there was a small but significant trend to detect more conflicts in longer reports (using total FCs as a measure of report length) ( Figure 1 ).
When conflict rate was analyzed by time of day of report finalization, there was a small but significant decrease in conflict rate as the day progressed. Conflicts per hour by time of day were as follows: 7 am to noon, 2.5 ± 2.1; noon to 5 pm , 2.4 ± 2.0; and 5 to 10 pm , 2.2 ± 1.9 ( P = .001). This variation was not explained by report length, as the average number of FCs per report remained unchanged throughout the day. This difference was also not accounted for by reading speed (studies finalized per hour), which tended to increase as the day went on: 7 am to noon, 2.6 ± 1.7; noon to 5 pm , 3.0 ± 2.0; and 5 to 10 pm , 3.5 ± 2.0 ( P = .001).
To evaluate the effect of reading load on conflict rate, the mean number of conflicts per report was assessed against the number of reports finalized by a single reader per hour time period. There was a significant linear increase in conflict rate as the number of reports finalized per hour increased from two to 10 ( Figure 2 ). This was driven primarily by an increase in errors requiring resolution (MR type). The increase in conflict rate with increased reading speed was not accounted for by the number of FCs per report, which remained steady despite the rate of reading.
Physicians resolved MR conflicts by adding or deleting FCs suggested by the tool (61%) or returning to the reports and making correction (39%). Overall, physicians ignored identified conflicts 52% of the time and altered their readings to eliminate the conflicts 48% of the time. MR conflicts could not be ignored, and SR conflicts were ignored 73% of the time. There was a significant difference among physicians in the frequency with which SR conflicts were ignored. This ignore rate varied from 32% to 89%. There was a strong positive correlation ( r = 0.87, P = .05) between physician conflict rate and frequency with which rules were ignored. Physicians who had more report conflicts were more likely to ignore those conflicts. Overall, at least one change was made in 54% of all finalized studies. Fifty-four percent of reports in which changes were made had single changes, 27% had two changes, and 19% had more than two changes. Survey results demonstrated that the physicians saw value in using the tool ( Table 3 ).
| Item | Mean (range) |
|---|---|
| The MR rules made my echo reports better ∗ | 4.0 (3–5) |
| The SR rules made my echo reports better ∗ | 3.8 (3–4) |
| The rules helped me avoid errors in report generation ∗ | 4.4 (4–5) |
| The rules helped me follow accepted guidelines for echo interpretation ∗ | 3.8 (3–4) |
| The tool help identify finding codes that absolutely conflict with each other † | 4.8 (4–5) |
| The tool help identify finding codes that appear to conflict with each other † | 4.0 (3–5) |
Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
