Adjusting trial results for biases in meta‐analysis: combining data‐based evidence on bias with detailed trial assessment

Summary Flaws in the conduct of randomized trials can lead to biased estimation of the intervention effect. Methods for adjustment of within‐trial biases in meta‐analysis include the use of empirical evidence from an external collection of meta‐analyses, and the use of expert opinion informed by the assessment of detailed trial information. Our aim is to present methods to combine these two approaches to gain the advantages of both. We make use of the risk of bias information that is routinely available in Cochrane reviews, by obtaining empirical distributions for the bias associated with particular bias profiles (combinations of risk of bias judgements). We propose three methods: a formal combination of empirical evidence and opinion in a Bayesian analysis; asking experts to give an opinion on bias informed by both summary trial information and a bias distribution from the empirical evidence, either numerically or by selecting areas of the empirical distribution. The methods are demonstrated through application to two example binary outcome meta‐analyses. Bias distributions based on opinion informed by trial information alone were most dispersed on average, and those based on opinions obtained by selecting areas of the empirical distribution were narrowest. Although the three methods for combining empirical evidence with opinion vary in ease and speed of implementation, they yielded similar results in the two examples.


Supporting information S1 WinBUGS code for adjusting trial results for total bias in meta-analysis
#Bayesian meta-analysis of case study meta-analysis A dataset (Ohlsson and Lacy 2004), incorporating an informative prior for total bias in each trial at high or unclear risk of bias for sequence generation, allocation concealment and/or blinding. )*100 #Percentage weight of trial in meta-analysis } d ~ dnorm(0,1.0E-5) #Prior for combined intervention effect (log odds ratio) p.tau2<-1/tau2 #Prior for between-trial variance in intervention effect tau2<-tau*tau tau~dunif(0,2) } #Data from 10 trials included in case study meta-analysis A and parameters for trial-specific bias distributions derived from method 3. The trials are ordered as in Figure 1 : 1=Sandberg 2000, 2=Clapp 1989, 3=Bussel 1990, 4=Fanaroff 1994, 5=Haque 1986, 6=Weisman 1994a, 7=Chirico 1987, 8=Conway 1990, 9=Ratrisawadi 1991, 10=Tanzer 1997 rC/nC, binary outcome data for control arm; rT/nT, binary outcome data for treatment arm; mu, mean of bias distribution; sigma, standard deviation of bias distribution; X, indicator of a high or unclear risk of bias judgement for at least one of sequence generation, allocation concealment and blinding.

S2.1 Statistical analysis of the ROBES data
We modelled the estimated intervention effect θim (log odds ratio) in trial i of meta-analysis m as: where ijm x are indicators of high or unclear risk of bias for design characteristic j (j=1 for sequence generation, j=2 for allocation concealment, j=3 for blinding). Hence trials at low risk of bias for all three characteristics are assumed to estimate im  . We assume a normal random-effects distribution across trials for these intervention effects: A hierarchical model was fitted to the trial-specific biases, which allowed the main effects ijm  to vary within each meta-analysis m and mean bias jm b to vary across meta-analyses m: The parameter j  estimates the average increase in between-trial heterogeneity among trials with a single high or unclear risk of bias judgement for design characteristic j, relative to those at low risk of bias for all three characteristics. The model assumes that average bias jm b is exchangeable across meta-analyses m with mean 0 j b , where exp( 0 j b ) is the average ratio of odds ratios (ROR) comparing the intervention effect in trials with one high or unclear risk of bias judgement for characteristic j to the intervention effect in trials with low risk of bias judgements for all three characteristics.
We allowed for interaction terms between pair-wise combinations of design characteristics.
Interaction terms ijm  (j=4, 5, 6) were assumed to have the same two-level structure as the main effects ijm  (j=1, 2, 3), with distinct variance components. The average bias in trials with more than one high or unclear risk of bias judgement (on the log odds ratio scale) is estimated as the sum of the coefficients representing the effects of individual characteristics and the coefficients representing the interaction terms involving these design characteristics. For example, for a trial at high or unclear risk of bias for sequence generation and allocation concealment and low risk of bias for blinding, the bias in intervention effect relative to a trial at low risk of bias for all characteristics is modelled as Within the full Bayesian model, we derived an empirical predictive distribution for total bias βnew expected in a new trial with each possible bias profile. For example, a predictive distribution for bias in a trial at high or unclear risk of bias for sequence generation and allocation concealment and low risk of bias for blinding is given by:  , ).
The derived predictive distributions for bias may serve as empirically-based prior distributions for total bias βi in each trial i in a new binary outcome meta-analysis, within the model for bias adjustment described in the section, "Adjusting for bias".

S2.2 Descriptions of the meta-analyses and trials on which the distributions are based
64 meta-analyses (866 trials) from the ROBES database that assessed subjectively measured outcomes.
All meta-analyses include at least 5 trials and are informative for bias due to inadequate or unclear sequence generation, allocation concealment and blinding.

Distribution of outcome types
Proportion of trials at unclear risk of bias

S2.3 Empirical distributions for trial-specific biases
For each combination of risk of bias judgements (bias profile), we report a predictive normal distribution for the bias expected in a new trial with that same bias profile relative to a trial at low risk of bias. These distributions were derived from analysis of the 64 binary outcome meta-analyses (866 trials) included in the ROBES database.  Sepsis (presence of clinical findings of sepsis plus positive blood cultures).

Notes
Experimental group I and II were combined, and entered as one experimental group in the meta-analysis.

Risk of bias assessment and summary of methods used (as described in paper)
Adequate sequence generation? 'Unclear' risk of bias "The infants matched for gestational age, sex, weight and history of prolonged rupture of fetal membrane were randomly allocated into 3 groups of 34 each." Groups were comparable at baseline.

Allocation concealment? 'Unclear' risk of bias
No details provided.

Blinding? 'High' risk of bias
No placebo was used.
"The drug was not given to group III (controlled group)." Incomplete outcome data addressed?
Infants (number not stated) who expired within 24 hours of life or required blood exchange transfusion were excluded from the study. In spite of these exclusions the number of patients in each group is identical (N=34).
Denominator for analysis was the same as the number randomised per group.
'Unclear' risk of bias

Assessment Bussel 1990
L Sequence generation This trial has a low risk of bias judgement for sequence generation, a high/unclear risk of bias judgement for allocation concealment and a low risk of bias judgement for blinding. Where do you expect the ratio of odds ratios (ROR) comparing this trial against trials at low risk of bias to lie?
Mark the expected ROR value by an X and the inter-quartile range (IQR) by two lines | on the axis below. Choose inter-quartile range limits such that you believe the true ROR is equally likely to lie inside rather than outside the range. Provide numerical values if you wish, which may fall outside the scale given.
Note that the axis is on the log ratio of odds ratios (log ROR) scale, to assist our analyses. We will round the ROR to the nearest 0.02 on the log ROR scale.

Assessment Fanaroff 1994
L Sequence generation Evidence from a set of other trials at low risk of bias for sequence generation and allocation concealment and high/unclear risk of bias for blinding suggests that the bias might lie in the distribution represented by the Figure below.

H/U Blinding
Given this information, where in this distribution do you expect the bias in this particular trial to lie?
Mark the expected ROR value by an X and the inter-quartile range (IQR) by two lines | on the x-axis below. Choose inter-quartile range limits such that you believe the true ROR is equally likely to lie inside rather than outside the range. Provide numerical values if you wish, which may fall outside the range of the x-axis. See below for an example.
Note that the x-axis is on the log ratio of odds ratios (log ROR) scale, to assist our analyses. We will round the ROR to the nearest 0.02 on the log ROR scale.

Assessment Weisman 1994a
H/U Sequence generation Evidence from a set of other trials at high/unclear risk of bias for sequence generation and allocation concealment and low risk of bias for blinding suggests that the bias might lie in the distribution represented by the Figure  Meta-analysis A: opinions on the extent of bias, elicited from four out of twelve assessors, using three different methods. Plots (i) and (ii) display elicited inter-quartile ranges for bias, plot (iii) shows selected areas of empirically-derived bias distributions. H/U/L denote high/unclear/low risk of bias.
NB: The four assessors are not the same for each trial.