Selective outcome reporting is common among published randomized trials and is often associated with the reporting of positive study findings. We investigated whether publication of study protocols in publicly accessible formats is associated with the reporting of positive findings. An extended version of the Cochrane highly sensitive search strategy was used to identify reports of randomized trials on cardiovascular disease that were published in December 2012 and indexed in PubMed by November 2013. Study characteristics and methodologic characteristics were extracted in duplicate. The Fisher’s exact test and multivariable logistic regression were used to compare characteristics between trials that reported a publicly accessible protocol and those that did not. One hundred ninety-one reports of cardiovascular randomized trials were identified, 23 (12%) of which reported an accessible protocol. Trials reporting an accessible protocol were significantly larger and more likely to report strong trial methods, including reporting a power calculation, attrition, and the use of an intention-to-treat analysis. Despite greater statistical power, trials reporting an accessible protocol were less likely to report positive findings after controlling for known confounders (odds ratio 0.35, 95% confidence interval 0.13 to 0.94). Reporting of an accessible protocol is associated with a reduced likelihood of reporting positive findings. Further investigation is needed to determine if this association is causal.
Selective outcome reporting is common among published reports of randomized controlled trials (RCTs). An analysis of published trials and their respective protocols submitted for ethical review found that 50% of efficacy outcomes and 65% of harm outcomes were incompletely reported. Furthermore, statistically significant outcomes were more likely to be fully reported than statistically nonsignificant outcomes. In a cross-sectional analysis of adverse event reporting, only 39% of randomized trials adequately reported clinical adverse events. An analysis of Cochrane systematic reviews found that 20% of Cochrane meta-analyses with a significant positive result became nonsignificant after adjustment for outcome reporting bias. However, no study, to our knowledge, have assessed whether studies that make their protocols publicly accessible are less likely to report significant positive findings. In theory, publication of protocols in a publicly accessible format could reduce selective outcome reporting. Unless modified post hoc, a publicly accessible protocols would increase accountability for the accurate and complete reporting of outcomes and enable readers to critically appraise published trials. Accordingly, we sought to quantify how often details of a publicly accessible protocol were reported within published reports of RCTs, in a cross-sectional analysis of cardiovascular RCTs published in December 2012. Furthermore, we assessed whether reporting of publicly accessible protocols was associated with study findings.
Methods
This study is part of an ongoing study on the epidemiology and methodologic conduct of published RCTs. All reports of RCTs in cardiovascular disease published in December 2012 and indexed in PubMed by November 17, 2013, were eligible for inclusion. We restricted our analysis to a cross section of randomized trials published in a single month as we wished to examine the association of reporting a publicly accessible protocol with study findings in the most recent time period available at the initiation of this study. A modified version of the Cochrane highly sensitive search strategy (phase 1 with added terms: “crossover studies” and “multicenter study”) was used to identify primary reports of RCTs. Randomized trials were defined as prospective studies that assessed health care interventions in human participants, or groups of participants, who were randomly allocated to study interventions. Cardiovascular randomized trials were defined as RCTs focused on a cardiovascular disease, as defined by the International Classification of Diseases, Tenth Revision . Only the primary publication of each randomized trial (i.e., that reporting the primary outcome) was eligible for inclusion; secondary publications and interim analyses were excluded. As our unit of analysis was at the trial level, multiple RCTs reported within the same publication were extracted independently.
Two investigators, CE and AO, reviewed abstracts, excluding studies that were not RCTs. Full-text articles were then obtained and reviewed in duplicate by the same reviewers. Of all eligible RCTs, the subgroup of studies examining cardiovascular diseases was specifically analyzed for this study. No language restrictions were applied; RCTs in Chinese, Spanish, French, and German were translated by AH, who has extensive experience in the translation of epidemiologic studies. Differences in eligibility and extraction (described subsequently) were resolved by referral to a third researcher (SH).
From each trial, 4 researchers (AO, CE, AH, and MS) extracted study and methodologic characteristics in duplicate. For study characteristics, the location of the corresponding author, the type of intervention, study design, number of study centers, number of arms, and sample size of the trial were recorded. For methodologic characteristics, whether the trial reported a power calculation, ≥1 primary outcomes, the method of random sequence generation and allocation concealment, method of blinding, who was blinded, use of an intention-to-treat analysis, or reported attrition was recorded (see Table 1 for definitions). Reporting of a publicly accessible protocol was defined as either the reporting of the trial protocol with a publication (often as a supplement) or reporting of a reference to the protocol in a publicly accessible journal or Web site. For each trial, 2 blinded reviewers also determined whether a significant positive, nonsignificant, or significant negative conclusion was reported, with differences resolved by consensus (see Table 1 for definitions).
| Items Assessed | Definitions |
|---|---|
| Study Findings | Identified as “significant positive” if an investigator defined primary outcome was statistically significant in favor of the experimental intervention. If the study was a non-inferiority trial, it was listed as “significant positive” if there was no significant difference between the experimental and control intervention. If no primary outcome was reported, we assessed the outcome that was emphasized in the abstract and study for statistical significance. |
| Power Calculation | Power calculation stated to have been undertaken: “Yes” or “No” |
| Primary Outcome | Primary outcome defined explicitly or an outcome used in power calculation, or a main outcome described explicitly in primary study objectives: “Yes” or “No” |
| Random Sequence Generation | Method used to allocate participants to study groups: “Computer”, “Random Number Table”, “Coin Toss”, “Not Reported or Inadequate”, “Other” |
| Allocation Concealment | Method used to prevent the individual enrolling participants from knowing or predicting the allocation sequence in advance: “Envelope”, “Central”, “Pharmacy”, “Not Reported or Inadequate”, “Other” |
| Blinding, How? | Method used to prevent participants, caregivers, investigators or outcomes assessors from knowing the intervention a participant received: “Blinded, details given (e.g. similar colour, taste)”, “Blinded, no details given (placebo)”, “Unblinded”, “Not Reported or Inadequate”. Note that trial is unblended if explicitly stated as such or blinding not possible. |
| Blinding, Who? | “Reported, details given (e.g. similar colour, taste)”, “Reported, no details given (placebo)”, “Unblinded”, “Unclear or Inadequate” |
| Attrition | Loss to follow up reported for each group: “Yes, details given”, “Yes, details not given”, “No” |
| Intention to Treat | Reported as having been analyzed in their assigned groups: “Intention to Treat”, “Not Intention to Treat or Inadquate” |
To determine whether study and methodologic characteristics differed between trials that reported a publicly accessible protocol and trials that did not report an accessible protocol, Fisher’s exact test was used. Multivariable logistic regression was also used to examine whether trials with accessible protocols were more likely to report positive study findings. Reporting of adequate blinding, method of randomization, allocation concealment, and size of trial were included in the model as possible confounders because of their association with exaggerated treatment effects in meta-epidemiologic studies. Institutional review board approval was not needed as this study did not involve human subjects.
Results
We identified 4,190 abstracts of possible reports of RCTs, of which 1,351 primary reports of RCTs were identified. Of these, a subgroup of 191 cardiovascular RCTs (14.1%) were identified ( Figure 1 ), of which 23 trials (12.0%) reported a publicly accessible protocol, whereas 168 (88.0%) did not report a protocol. The median impact factor of the journals that the trials included in this study were published in was 3.5 (interquartile range [IQR] 2.3 to 6.5). Overall, 119 trials reported significant positive findings, 61 reported nonsignificant findings, and 11 reported negative findings.
Trials that reported an accessible protocol differed from trials that did not report a protocol ( Table 2 ). Protocol-reporting trials were significantly more likely to involve multiple centers and define a primary outcome. The median sample size of a protocol-reporting trial was significantly larger than trials that did not report an accessible protocol (262, interquartile range 78 to 1,433 vs 62, IQR 30 to 148, p <0.001). Protocol-reporting trials were more likely to report a power calculation, report centralized allocation concealment, report an intention-to-treat analysis, and adequately describe attrition (p <0.05 for all comparisons, Table 2 ).
| Accessible Protocol Reported? | p-value | ||
|---|---|---|---|
| No n=168 (%) | Yes n= 23 (%) | ||
| Study Characteristics | |||
| Cardiovascular Disease or Risk Factor under Study | |||
| Coronary heart disease | 55 (32.7%) | 6 (26%) | 0.14 |
| Ischemic stroke | 18 (10.7%) | 1 (4%) | |
| Hypertension | 14 (8.3%) | 3 (13%) | |
| Cardiomyopathy | 7 (4.2%) | 4 (17%) | |
| Peripheral vascular disease | 11 (6.6%) | 0 (0%) | |
| Other cardiovascular diseases/risk factors | 63 (37.5%) | 9 (39%) | |
| Location (Corresponding author) | |||
| High income country | 139 (82.7%) | 21 (91%) | 0.38 |
| Low or middle income country | 29 (17.3%) | 2 (9%) | |
| Intervention | |||
| Drug (including placebo) | 73 (43.5%) | 9 (39%) | 0.68 |
| Behavioral/Lifestyle/Education/Counseling | 30 (17.9%) | 4 (17%) | |
| Device (including sham) | 19 (11.3%) | 4 (17%) | |
| Dietary Supplement (e.g., vitamins, minerals) | 16 (9.5%) | 1 (4%) | |
| Genetic (including gene transfer, stem cell and recombinant DNA) | 2 (1.2%) | 1 (4%) | |
| Procedure/Surgery | 22 (13.1%) | 4 (17%) | |
| Other | 6 (3.6%) | 0 (0%) | |
| Study Centers | |||
| Multiple | 52 (31%) | 16 (70%) | 0.002 |
| Single | 57 (33.9%) | 4 (17%) | |
| Unclear | 59 (35.1%) | 3 (13%) | |
| Design | |||
| Parallel-Group | 128 (76.2%) | 18 (78%) | 0.08 |
| Cluster | 1 (0.6%) | 1 (4%) | |
| Crossover | 38 (22.6%) | 3 (13%) | |
| Factorial | 1 (0.6%) | 1 (4%) | |
| Number of Arms | |||
| 2 | 131 (78%) | 17 (74%) | 0.85 |
| 3 | 23 (13.7%) | 4 (17%) | |
| 4 | 9 (5.4%) | 1 (4%) | |
| >4 | 5 (3%) | 1 (4%) | |
| Outcome Type | |||
| Surrogate | 112 (66.7%) | 12 (52%) | 0.13 |
| Clinical | 56 (33.3%) | 11 (48%) | |
| Sample Size | |||
| Median (Interquartile range) | 62 (30, 148) | 262 (78, 1433) | <0.001 |
| Outcome Direction | |||
| Significant positive | 111 (66.1%) | 8 (35%) | 0.001 |
| Nonsignificant | 51 (30.4%) | 10 (44%) | |
| Significant negative | 6 (3.6%) | 5 (22%) | |
| Methodological Characteristics | |||
| Defined Primary Outcome | |||
| Yes | 115 (68.5%) | 22 (96%) | 0.005 |
| No | 53 (31.5%) | 1 (4%) | |
| Power Calculation | |||
| Yes | 87 (51.8%) | 17 (74%) | 0.07 |
| No | 81 (48.2%) | 6 (26%) | |
| Random Sequence Generation | |||
| Computer | 57 (33.9%) | 11 (48%) | 0.56 |
| Not Reported | 98 (58.3%) | 12 (52%) | |
| Other | 7 (4.2%) | 0 (0%) | |
| Random Number Table | 6 (3.6%) | 0 (0%) | |
| Blinding, How? | |||
| Blinded, details given (e.g. similar color, taste) | 30 (17.9%) | 7 (30%) | 0.10 |
| Blinded, no details given (e.g. placebo) | 85 (50.6%) | 7 (30%) | |
| Not Reported | 36 (21.4%) | 4 (17%) | |
| Unblinded | 17 (10.1%) | 5 (22%) | |
| Blinding, Who? | |||
| Reported, details given (e.g. pt, caregiver) | 77 (45.8%) | 8 (35%) | 0.37 |
| Reported, no details (e.g. double blind) | 38 (22.6%) | 6 (26%) | |
| Unblinded | 17 (10.1%) | 5 (22%) | |
| Unclear | 36 (21.4%) | 4 (17%) | |
| Allocation Concealment | |||
| Central | 9 (5.4%) | 6 (26%) | 0.001 |
| Envelope | 43 (25.6%) | 0 (0%) | |
| Other | 4 (2.4%) | 1 (4%) | |
| Pharmacy | 8 (4.8%) | 2 (9%) | |
| Not Reported | 104 (61.9%) | 14 (61%) | |
| Attrition | |||
| No | 72 (42.9%) | 2 (9%) | 0.003 |
| Yes, details given | 84 (50%) | 19 (83%) | |
| Yes, details not given | 12 (7.1%) | 2 (9%) | |
| Intention to Treat | |||
| Yes | 29 (17.3%) | 10 (44%) | 0.01 |
| No | 139 (82.7%) | 13 (57%) | |
Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
