Reporting of a Publicly Accessible Protocol and Its Association With Positive Study Findings in Cardiovascular Trials (from the Epidemiological Study of Randomized Trials [ESORT])

Selective outcome reporting is common among published randomized trials and is often associated with the reporting of positive study findings. We investigated whether publication of study protocols in publicly accessible formats is associated with the reporting of positive findings. An extended version of the Cochrane highly sensitive search strategy was used to identify reports of randomized trials on cardiovascular disease that were published in December 2012 and indexed in PubMed by November 2013. Study characteristics and methodologic characteristics were extracted in duplicate. The Fisher’s exact test and multivariable logistic regression were used to compare characteristics between trials that reported a publicly accessible protocol and those that did not. One hundred ninety-one reports of cardiovascular randomized trials were identified, 23 (12%) of which reported an accessible protocol. Trials reporting an accessible protocol were significantly larger and more likely to report strong trial methods, including reporting a power calculation, attrition, and the use of an intention-to-treat analysis. Despite greater statistical power, trials reporting an accessible protocol were less likely to report positive findings after controlling for known confounders (odds ratio 0.35, 95% confidence interval 0.13 to 0.94). Reporting of an accessible protocol is associated with a reduced likelihood of reporting positive findings. Further investigation is needed to determine if this association is causal.

Selective outcome reporting is common among published reports of randomized controlled trials (RCTs). An analysis of published trials and their respective protocols submitted for ethical review found that 50% of efficacy outcomes and 65% of harm outcomes were incompletely reported. Furthermore, statistically significant outcomes were more likely to be fully reported than statistically nonsignificant outcomes. In a cross-sectional analysis of adverse event reporting, only 39% of randomized trials adequately reported clinical adverse events. An analysis of Cochrane systematic reviews found that 20% of Cochrane meta-analyses with a significant positive result became nonsignificant after adjustment for outcome reporting bias. However, no study, to our knowledge, have assessed whether studies that make their protocols publicly accessible are less likely to report significant positive findings. In theory, publication of protocols in a publicly accessible format could reduce selective outcome reporting. Unless modified post hoc, a publicly accessible protocols would increase accountability for the accurate and complete reporting of outcomes and enable readers to critically appraise published trials. Accordingly, we sought to quantify how often details of a publicly accessible protocol were reported within published reports of RCTs, in a cross-sectional analysis of cardiovascular RCTs published in December 2012. Furthermore, we assessed whether reporting of publicly accessible protocols was associated with study findings.


This study is part of an ongoing study on the epidemiology and methodologic conduct of published RCTs. All reports of RCTs in cardiovascular disease published in December 2012 and indexed in PubMed by November 17, 2013, were eligible for inclusion. We restricted our analysis to a cross section of randomized trials published in a single month as we wished to examine the association of reporting a publicly accessible protocol with study findings in the most recent time period available at the initiation of this study. A modified version of the Cochrane highly sensitive search strategy (phase 1 with added terms: “crossover studies” and “multicenter study”) was used to identify primary reports of RCTs. Randomized trials were defined as prospective studies that assessed health care interventions in human participants, or groups of participants, who were randomly allocated to study interventions. Cardiovascular randomized trials were defined as RCTs focused on a cardiovascular disease, as defined by the International Classification of Diseases, Tenth Revision . Only the primary publication of each randomized trial (i.e., that reporting the primary outcome) was eligible for inclusion; secondary publications and interim analyses were excluded. As our unit of analysis was at the trial level, multiple RCTs reported within the same publication were extracted independently.

Two investigators, CE and AO, reviewed abstracts, excluding studies that were not RCTs. Full-text articles were then obtained and reviewed in duplicate by the same reviewers. Of all eligible RCTs, the subgroup of studies examining cardiovascular diseases was specifically analyzed for this study. No language restrictions were applied; RCTs in Chinese, Spanish, French, and German were translated by AH, who has extensive experience in the translation of epidemiologic studies. Differences in eligibility and extraction (described subsequently) were resolved by referral to a third researcher (SH).

From each trial, 4 researchers (AO, CE, AH, and MS) extracted study and methodologic characteristics in duplicate. For study characteristics, the location of the corresponding author, the type of intervention, study design, number of study centers, number of arms, and sample size of the trial were recorded. For methodologic characteristics, whether the trial reported a power calculation, ≥1 primary outcomes, the method of random sequence generation and allocation concealment, method of blinding, who was blinded, use of an intention-to-treat analysis, or reported attrition was recorded (see Table 1 for definitions). Reporting of a publicly accessible protocol was defined as either the reporting of the trial protocol with a publication (often as a supplement) or reporting of a reference to the protocol in a publicly accessible journal or Web site. For each trial, 2 blinded reviewers also determined whether a significant positive, nonsignificant, or significant negative conclusion was reported, with differences resolved by consensus (see Table 1 for definitions).

Table 1

Definitions used to assess reporting of methodological details in publications of randomized trials

Items Assessed Definitions
Study Findings Identified as “significant positive” if an investigator defined primary outcome was statistically significant in favor of the experimental intervention. If the study was a non-inferiority trial, it was listed as “significant positive” if there was no significant difference between the experimental and control intervention. If no primary outcome was reported, we assessed the outcome that was emphasized in the abstract and study for statistical significance.
Power Calculation Power calculation stated to have been undertaken: “Yes” or “No”
Primary Outcome Primary outcome defined explicitly or an outcome used in power calculation, or a main outcome described explicitly in primary study objectives: “Yes” or “No”
Random Sequence Generation Method used to allocate participants to study groups: “Computer”, “Random Number Table”, “Coin Toss”, “Not Reported or Inadequate”, “Other”
Allocation Concealment Method used to prevent the individual enrolling participants from knowing or predicting the allocation sequence in advance: “Envelope”, “Central”, “Pharmacy”, “Not Reported or Inadequate”, “Other”
Blinding, How? Method used to prevent participants, caregivers, investigators or outcomes assessors from knowing the intervention a participant received: “Blinded, details given (e.g. similar colour, taste)”, “Blinded, no details given (placebo)”, “Unblinded”, “Not Reported or Inadequate”. Note that trial is unblended if explicitly stated as such or blinding not possible.
Blinding, Who? “Reported, details given (e.g. similar colour, taste)”, “Reported, no details given (placebo)”, “Unblinded”, “Unclear or Inadequate”
Attrition Loss to follow up reported for each group: “Yes, details given”, “Yes, details not given”, “No”
Intention to Treat Reported as having been analyzed in their assigned groups: “Intention to Treat”, “Not Intention to Treat or Inadquate”

To determine whether study and methodologic characteristics differed between trials that reported a publicly accessible protocol and trials that did not report an accessible protocol, Fisher’s exact test was used. Multivariable logistic regression was also used to examine whether trials with accessible protocols were more likely to report positive study findings. Reporting of adequate blinding, method of randomization, allocation concealment, and size of trial were included in the model as possible confounders because of their association with exaggerated treatment effects in meta-epidemiologic studies. Institutional review board approval was not needed as this study did not involve human subjects.


We identified 4,190 abstracts of possible reports of RCTs, of which 1,351 primary reports of RCTs were identified. Of these, a subgroup of 191 cardiovascular RCTs (14.1%) were identified ( Figure 1 ), of which 23 trials (12.0%) reported a publicly accessible protocol, whereas 168 (88.0%) did not report a protocol. The median impact factor of the journals that the trials included in this study were published in was 3.5 (interquartile range [IQR] 2.3 to 6.5). Overall, 119 trials reported significant positive findings, 61 reported nonsignificant findings, and 11 reported negative findings.

Figure 1

Identification of cardiovascular trials for inclusion.

Trials that reported an accessible protocol differed from trials that did not report a protocol ( Table 2 ). Protocol-reporting trials were significantly more likely to involve multiple centers and define a primary outcome. The median sample size of a protocol-reporting trial was significantly larger than trials that did not report an accessible protocol (262, interquartile range 78 to 1,433 vs 62, IQR 30 to 148, p <0.001). Protocol-reporting trials were more likely to report a power calculation, report centralized allocation concealment, report an intention-to-treat analysis, and adequately describe attrition (p <0.05 for all comparisons, Table 2 ).

Table 2

Reported epidemiological characteristics of cardiovascular trials

Accessible Protocol Reported? p-value
n=168 (%)
n= 23 (%)
Study Characteristics
Cardiovascular Disease or Risk Factor under Study
Coronary heart disease 55 (32.7%) 6 (26%) 0.14
Ischemic stroke 18 (10.7%) 1 (4%)
Hypertension 14 (8.3%) 3 (13%)
Cardiomyopathy 7 (4.2%) 4 (17%)
Peripheral vascular disease 11 (6.6%) 0 (0%)
Other cardiovascular diseases/risk factors 63 (37.5%) 9 (39%)
Location (Corresponding author)
High income country 139 (82.7%) 21 (91%) 0.38
Low or middle income country 29 (17.3%) 2 (9%)
Drug (including placebo) 73 (43.5%) 9 (39%) 0.68
Behavioral/Lifestyle/Education/Counseling 30 (17.9%) 4 (17%)
Device (including sham) 19 (11.3%) 4 (17%)
Dietary Supplement (e.g., vitamins, minerals) 16 (9.5%) 1 (4%)
Genetic (including gene transfer, stem cell and recombinant DNA) 2 (1.2%) 1 (4%)
Procedure/Surgery 22 (13.1%) 4 (17%)
Other 6 (3.6%) 0 (0%)
Study Centers
Multiple 52 (31%) 16 (70%) 0.002
Single 57 (33.9%) 4 (17%)
Unclear 59 (35.1%) 3 (13%)
Parallel-Group 128 (76.2%) 18 (78%) 0.08
Cluster 1 (0.6%) 1 (4%)
Crossover 38 (22.6%) 3 (13%)
Factorial 1 (0.6%) 1 (4%)
Number of Arms
2 131 (78%) 17 (74%) 0.85
3 23 (13.7%) 4 (17%)
4 9 (5.4%) 1 (4%)
>4 5 (3%) 1 (4%)
Outcome Type
Surrogate 112 (66.7%) 12 (52%) 0.13
Clinical 56 (33.3%) 11 (48%)
Sample Size
Median (Interquartile range) 62 (30, 148) 262 (78, 1433) <0.001
Outcome Direction
Significant positive 111 (66.1%) 8 (35%) 0.001
Nonsignificant 51 (30.4%) 10 (44%)
Significant negative 6 (3.6%) 5 (22%)
Methodological Characteristics
Defined Primary Outcome
Yes 115 (68.5%) 22 (96%) 0.005
No 53 (31.5%) 1 (4%)
Power Calculation
Yes 87 (51.8%) 17 (74%) 0.07
No 81 (48.2%) 6 (26%)
Random Sequence Generation
Computer 57 (33.9%) 11 (48%) 0.56
Not Reported 98 (58.3%) 12 (52%)
Other 7 (4.2%) 0 (0%)
Random Number Table 6 (3.6%) 0 (0%)
Blinding, How?
Blinded, details given (e.g. similar color, taste) 30 (17.9%) 7 (30%) 0.10
Blinded, no details given (e.g. placebo) 85 (50.6%) 7 (30%)
Not Reported 36 (21.4%) 4 (17%)
Unblinded 17 (10.1%) 5 (22%)
Blinding, Who?
Reported, details given (e.g. pt, caregiver) 77 (45.8%) 8 (35%) 0.37
Reported, no details (e.g. double blind) 38 (22.6%) 6 (26%)
Unblinded 17 (10.1%) 5 (22%)
Unclear 36 (21.4%) 4 (17%)
Allocation Concealment
Central 9 (5.4%) 6 (26%) 0.001
Envelope 43 (25.6%) 0 (0%)
Other 4 (2.4%) 1 (4%)
Pharmacy 8 (4.8%) 2 (9%)
Not Reported 104 (61.9%) 14 (61%)
No 72 (42.9%) 2 (9%) 0.003
Yes, details given 84 (50%) 19 (83%)
Yes, details not given 12 (7.1%) 2 (9%)
Intention to Treat
Yes 29 (17.3%) 10 (44%) 0.01
No 139 (82.7%) 13 (57%)

Only gold members can continue reading. Log In or Register to continue

Stay updated, free articles. Join our Telegram channel

Nov 28, 2016 | Posted by in CARDIOLOGY | Comments Off on Reporting of a Publicly Accessible Protocol and Its Association With Positive Study Findings in Cardiovascular Trials (from the Epidemiological Study of Randomized Trials [ESORT])

Full access? Get Clinical Tree

Get Clinical Tree app for offline access