Proper randomization and adequate allocation concealment, which prevents clinicians and participants becoming aware of upcoming assignments are important strategies to protect from bias. Inadequate allocation concealment leads to exaggerated estimates of treatment effect [9]. Major limitations in study design may lead to rating down the quality of evidence for an outcome. However, assessment of whether or not a methodological shortcoming, such as lack of blinding, may have had a substantial impact on an estimate of effect is important as there are situations where lack of blinding may not materially impact a particular outcome. Another issue that is commonly encountered with RCTs is losses to follow up. Again, losses to follow up may not always require rating down if there are few and proportionate losses to follow up in both treatment and control groups. However, disproportionate losses to follow up can either increase (due to greater losses in the control group) or decrease (due to greater losses in the treatment group) the treatment effect [10]. The way in which RCTs are analyzed is another important criterion to consider in study design. Intention-to-treat (ITT) analysis is the preferred method of analysis of RCTs. However, it is documented that the intention-to-treat approach is often inadequately described and inadequately applied in RCT and deviations from ITT analysis are common [11]. RCTs should be carefully reviewed to determine if they adopted the ITT approach for a particular outcome. Lastly, authors of systematic reviews and guideline developers should exercise caution when they encounter trials that are stopped early for benefit, particularly when such trials contribute considerable weight to a meta-analysis as they might produce a spurious improvement in the treatment effect [12, 13].

Confidence in the estimate of effect may require rating down for inconsistency, if the magnitude and direction of effects across different studies varies widely (heterogeneity of study results). Variability in treatment effects across studies usually is the result of varying populations or interventions. However, when the reasons for inconsistency across studies cannot be identified, the confidence in the evidence may be lower. Consider for example the effect of suction vs. non-suction on prolonged air leakage to the underwater seal drains following pulmonary surgery. A meta-analysis of available RCTs showed varying effect estimates and direction of effect resulting in an I-squared of residual heterogeneity of close to 60 %, which could be considered substantial and it would not be unreasonable to rate down for inconsistency (Fig. 2.2) [14].

It is particularly important to remember that in GRADE, the quality of evidence is not rated up for consistency, it is only rated down for inconsistency. Several criteria may help decide whether heterogeneity exists: the point estimates vary widely across studies; minimally or non-overlapping confidence intervals; statistical test for heterogeneity shows a low p-value; I-squared value (percentage of variability due to heterogeneity rather than chance) is large [15].

GRADE defines several sources of indirectness. For example, differences in patient characteristics (age, gender and race), differences in interventions or comparators (similar but not the same intervention or comparators), indirectness of outcomes (direct outcome measures vs. surrogate outcome measures) and indirect comparisons (e.g., lack of head-to-head trials of competing surgical approaches). All sources of indirectness can result in lowering our confidence in the estimate of effects. However, it is necessary to remember that when direct evidence is limited in quantity or quality, indirect evidence from other populations may be considered and the quality need not necessarily be rated down with proper justification for not doing so. For example, although direct evidence about the safety and effectiveness of prophylaxis of VTE prevention in patients undergoing thoracic surgery is limited, the ACCP anti-thrombotic guidelines did not rate down for indirectness as they felt that the evidence about relative risks from studies of patients undergoing general or abdominal-pelvic surgery could be applied with little or no indirectness to thoracic surgery [16]. Another domain of indirectness is duration of follow-up for certain outcomes. GRADE recommends that guideline developers should always indicate the length of follow up to which the estimate of absolute effect refers. This length of follow up is a time frame judged appropriate to balance the risk-benefit consequences of alternative treatment strategies. Longer follow up periods are associated with higher risk differences between intervention and control. This could potentially lead to important differences in readers’ perception of the apparent magnitude of effect. Often, extending the time frame involves the assumption that event rates will stay constant over time [17].

Of particular importance is the categorization of outcome measures into direct and surrogate outcomes. In the absence of data on patient-important outcomes, surrogates could contribute to the estimation of the effect of an intervention on the outcomes that are important. Post-surgical asymptomatic deep vein thrombosis detected by screening venography or ultrasound surveillance is an example of a surrogate outcome [18]. It is to be noted that despite the relative importance of direct outcomes, both direct and surrogate outcomes should be reported in studies because the audience for guideline developers and systematic reviews might want to see both before making appropriate decisions.

Imprecision is usually determined by examining the confidence intervals. Usually, studies with few enrolled patients and/or few events have wider confidence intervals. Additionally, our confidence in the evidence is lowered when the 95 % confidence interval fails to exclude important benefit or important harm. Consider for example the long-term outcome of dilation requirements when using 180-degree laparoscopic anterior fundoplication (180-degree LAF) versus laparoscopic Nissen fundoplication (LNF) for GERD [19]. Although the partial fundoplication showed less than half the rate of dilatations, few events in the studies and generally low sample sizes did not allow for a precise estimate even after pooling the results, and the 95 % confidence interval crosses one (Fig. 2.3).