Atkins and colleagues' meta-analysis [1] of stroke in randomized controlled trials of cholesterol reduction shows the problems inherent in this technique. The first is publication bias; unpublished studies may have systematically different results from published ones. The second is selection bias; required data may be unavailable in published sources. For example, stroke mortality data were only reported for a few single-factor cholesterol intervention studies, and only such trials were included. Researchers doing meta-analyses should obtain the most complete data possible. We collected these data while doing a recent meta-analysis of cholesterol lowering and mortality [2]. For all trials of nonhormonal treatments, including trials with female participants [2, 3], the pooled fixed-effects odds ratio for stroke mortality in the treatment arms against the control arms is 1.10 (95% CI, 0.82 to 1.48), compared with 1.34 (95% CI, 0.91 to 1.96) reported by Atkins and colleagues. The former analysis is based on 66% more deaths from stroke than the latter, as reflected in the narrower confidence interval.

Third, the correct type of data must be collected. In particular, the results from an intention-to-treat analysis should be obtained whenever possible. This was not the case in the second most highly weighted trial included by Atkins and colleagues. In the initial reports from the World Health Organization clofibrate trial [4], results were presented for participants who remained "in trial." Eleven deaths from stroke were reported, with an odds ratio of 1.74 (0.5 to 7.1) between treatment and control groups. Since then, the investigators have published the data on an intention-to-treat basis [5]. With 25 deaths from stroke the odds ratio is now 1.27 (95% CI, 0.5 to 3.0). The latter data should have been used rather than the former.

Fourth, subgroup analyses should be treated with caution. An analysis of three trials "using clofibrate" was done, which yielded an impressive odds ratio of 2.64 (95% CI, 1.42 to 4.92) for fatal stroke. This analysis, however, included inappropriate data from the World Health Organization trial and the Stockholm study of combined niacin and clofibrate therapy, while excluding four other clofibrate trials. An analysis of the single-factor clofibrate trials using the correct data yields an odds ratio of 1.40 (95% CI, 0.88 to 2.21). When multiple subgroup analyses are done, it is not surprising that some yield striking results. For example, an analysis of trials for which the first author's name begins with "D" yields an odds ratio of 0.34 (95% CI, 0.13 to 0.88). However, one would not advise enrolling into a trial with a principal investigator whose name begins with "D" as a way of avoiding stroke.

Inconsistencies among the findings of meta-analyses are generally caused by different criteria for data inclusion. For example, it is not obvious why data from female participants were excluded. It is also unclear how it was decided that in the Stockholm study, "end points had been assessed without knowledge of treatment group" because this information appears in none of the papers that we have consulted. Rather than attempting to construct an optimal set of inclusion criteria for a meta-analysis, a thorough sensitivity analysis should be done that excludes studies of poor quality, small studies, single studies with a large influence, and so forth. Only findings robust enough to withstand such analysis should be considered substantiated. None of the substantive findings of the analysis by Atkins and colleagues would survive such a sensitivity analysis. Without sufficient data, meta-analysis can cause more confusion than enlightenment.