Systematic Reviews: Synthesis of Best Evidence for Clinical Decisions

Incorporating Research Evidence into Clinical Decision Making

The foregoing scenario is familiar to practitioners. In a typical week, we encounter patients with diverse problems; exercise numerous clinical, interpersonal, and technical skills; and make many decisions. The factors that affect these decisions and their outcomes are complex. For instance, each patient has unique sociodemographic characteristics, cultural circumstances, and personal preferences. Each physician has unique knowledge, experiences, and values. Moreover, practitioners and their patients make decisions within the context of a rapidly changing health care system that influences the availability, accessibility, and cost of diagnostic tests and therapies [6].

Timely, useful evidence from the biomedical literature should be an integral component of clinical decision making. If one treatment has been shown to be better than another, we need to know, so that we can recommend the treatment to the appropriate patients. The worldwide effort to develop new tests and treatments, and to determine their usefulness, has never been stronger, and our patients and their families expect us to be fonts of the knowledge that results from this effort [7]. Unfortunately, it is easy for current best research evidence to pass us by [8]. We may lack the time, motivation, and basic skills needed to find, critically appraise, and synthesize information, all of which we must do if we are to integrate the results of original studies into our practice. Fortunately, several potent methods are emerging that can greatly enhance our ability to interpret and apply research evidence; foremost among them is the systematic review.

This article begins a series in Annals that will examine systematic reviews in detail and explore their many applications. Systematic reviews represent the best chance that most practitioners will have to understand and accurately apply the key signals arising from the robust and increasingly productive search for solutions to medical problems. A properly conducted systematic review faithfully summarizes the evidence from all relevant studies on the topic of interest, and it does so concisely and transparently.

What Is a Systematic Review?

Systematic reviews are scientific investigations in themselves, with pre-planned methods and an assembly of original studies as their “subjects.” They synthesize the results of multiple primary investigations by using strategies that limit bias and random error [9, 10]. These strategies include a comprehensive search of all potentially relevant articles and the use of explicit, reproducible criteria in the selection of articles for review. Primary research designs and study characteristics are appraised, data are synthesized, and results are interpreted.

When the results of primary studies are summarized but not statistically combined, the review may be called a qualitative systematic review. A quantitative systematic review, or meta-analysis, is a systematic review that uses statistical methods to combine the results of two or more studies. The term “overview” is sometimes used to denote a systematic review, whether quantitative or qualitative. Summaries of research that lack explicit descriptions of systematic methods are often called narrative reviews.

Review articles are one type of integrative publication; practice guidelines, economic evaluations, and clinical decision analyses are others. These other types of integrative articles often incorporate the results of systematic reviews. For example, practice guidelines are systematically developed statements intended to assist practitioners and patients with decisions about appropriate health care for specific clinical circumstances [11]. Evidence-based practice guidelines are based on systematic reviews of the literature, appropriately adapted to local circumstances and values. Economic evaluations compare both the costs and the consequences of different courses of action; the knowledge of consequences that are considered in these evaluations is often generated by systematic reviews of primary studies. Decision analyses quantify both the likelihood and the valuation of the expected outcomes associated with competing alternatives.

Differences between Systematic and Narrative Reviews

All reviews, narrative and systematic alike, are retrospective, observational research studies and are therefore subject to systematic and random error. Accordingly, the quality of a review-and thus its worth-depends on the extent to which scientific review methods have been used to minimize error and bias. This is the key feature that distinguishes traditional narrative reviews from systematic reviews (Table 1). If a review is prepared according to the steps outlined in the right column of Table 1, it is more likely to be systematic and to provide unbiased conclusions. If review methods approximate those found in the middle column of Table 1, the article is more likely to be a narrative review, and the conclusions are less likely to be based on an unbiased summary of all relevant evidence.

Systematic reviews are generated to answer specific, often narrow, clinical questions in depth. These questions can be formulated explicitly according to four variables: a specific population and setting (such as elderly outpatients), the condition of interest (for example, hypertension), an exposure to a test or treatment (such as pharmacologic management), and one or more specific outcomes (such as cardiovascular and cerebrovascular events and mortality) [12]. Thus, an example of a well-formulated, clinically relevant question is, Does pharmacologic treatment of hypertension in the elderly prevent strokes and myocardial infarctions or delay death? If the question that is driving the review is not clear from the title, abstract, or introduction, or if no methods section is included, the paper is more likely to be a narrative review than a systematic review [13].

Most narrative review articles deal with a broad range of issues related to a given topic rather than addressing a particular issue in depth [9]. For example, a narrative review on diabetes (such as that which might be found in a textbook chapter) might include sections on the physiology and pathophysiology of carbohydrate, lipid, and protein metabolism; the epidemiology of and prognosis associated with diabetes; diagnostic and screening approaches; and preventive, therapeutic, rehabilitative, and palliative interventions. Thus, narrative reviews may be most useful for obtaining a broad perspective on a topic; they are less often useful in furnishing quantitative answers to specific clinical questions.

Narrative reviews are appropriate for describing the history or development of a problem and its management. Narrative reviews may better describe cutting-edge developments if research is scant or preliminary or if studies are very limited by flawed design or execution [13]. They may be particularly useful for discussing data in light of underlying theory and context. Narrative reviews can draw analogies and can conceptually integrate two independent fields of research, such as cancer and the acquired immunodeficiency syndrome [13].

However, the connection between clinical recommendations and evidence in narrative reviews is often tenuous, incomplete, or-worse still-based on a biased citation of studies [14, 15]. As a result, recommendations found in narrative reviews published in journals and textbooks often differ from recommendations found in systematic reviews. For example, narrative reviews may lag behind by more than a decade in endorsing a treatment of proven effectiveness, or they may continue to advocate a therapy long after it has been shown to be useless or harmful [16]. Also, systematic reviews that incorporate quantitative techniques are more likely than narrative reviews to detect small but clinically meaningful treatment effects [17].

What Systematic Reviews Can and Cannot Do

A well-conducted systematic review is invaluable for practitioners. Many of us feel overwhelmed by the volume of medical literature and, as a result, often prefer summaries of information to publications of original investigations [18]. Thus, review articles can help us keep up-to-date. High-quality systematic reviews can define the boundaries of what is known and what is not known and can help us avoid knowing less than has been proven.

It is unusual for single studies to provide definitive answers to clinical questions [19], but systematic reviews can help practitioners solve specific clinical problems. By critically examining primary studies, systematic reviews can also improve our understanding of inconsistencies among diverse pieces of research evidence. By quantitatively combining the results of several small studies, meta-analyses can create more precise, powerful, and convincing conclusions. An example of this is the recent review highlighting the beneficial effect of dietary protein restriction on the progression of diabetic and nondiabetic renal disease [20]. In addition, systematic reviews of several studies may better inform us about whether findings can be applied to specific subgroups of patients [21].

Investigators need systematic reviews to summarize existing data, refine hypotheses, estimate sample sizes, and help define future research agendas. Without systematic reviews, researchers may miss promising leads or may embark on studies of questions that have been already answered. Administrators and purchasers need review articles and other integrative publications to help generate clinical policies that optimize outcomes using available resources [22].

Systematic reviews can aid, but can never replace, sound clinical reasoning. Clinicians reason about individual patients on the basis of analogy, experience, heuristics, and theory as well as research evidence [23-25]. Awareness of a treatment's effectiveness does not confer knowledge about how to use that treatment in caring for individual patients. Evidence can lead to bad practice if it is applied in an uncritical or unfeeling way [25]. Understanding the complex structure of medical decision making [26] requires an appreciation of the ways in which knowledge, skills, values, and research evidence are integrated in each patient-clinician encounter.

The Past, Present, and Future of Systematic Reviews

The epidemiology of review articles is changing rapidly; the number of systematic reviews published annually has increased at least 500-fold in the past decade [27, 28]. It is not unusual now to find more than one systematic review addressing the same or similar therapeutic questions [29]. One example is the evaluation of the effect of calcium on blood pressure, which was summarized in two recent meta-analyses published within 1 month of each other [30, 31]. Although duplicate independent reviews that generate similar results may increase our confidence in those results, multiple reviews with similar results may have an opportunity cost-they may divert effort away from the development of systematic reviews in other areas where they may be needed. The situation can become more confusing when several reviews yield similar results but present different clinical recommendations; this occurred when five meta-analyses of selective decontamination of the digestive tract in critically ill patients were published within 5 years (two were published in the same issue of the same journal) [32-36]. However, resolving discord between meta-analyses can be a cooperative, informative exercise that clarifies review methods and synthesizes all currently available evidence [37].

In the past 10 years, guides have been created to aid in the critical appraisal [38, 39] and application [40] of review articles. A framework for interpreting primary research and systematic reviews by using levels of evidence and grades of recommendations [41] has been successfully employed by several groups. At the Antithrombotic Consensus Conferences of the American College of Chest Physicians, therapeutic recommendations are routinely classified according to their evidentiary base-whether they are predicated on the results of large, rigorous, randomized trials or on meta-analyses, observational studies, or expert opinion [41].

An international initiative called the Cochrane Collaboration has evolved to help prepare, maintain, and disseminate the results of systematic reviews of health care interventions [42, 43]. The Cochrane Library is the first large-scale, multidisciplinary product of this collaboration. It is updated quarterly, and in January 1997 it contained 159 completed systematic reviews and 199 systematic reviews in preparation from the Cochrane Collaboration itself, a bibliography of more than 1600 other systematic review abstracts in the medical literature, a register of controlled trials that included slightly more than 110 000 trials, and a bibliography of 400 methodologic articles about the science of reviewing research [44].

What does the future hold for systematic reviews? The coverage of medical topics by systematic reviews, although expanding rapidly, is still limited. The Cochrane Collaboration is facilitating this mission but needs increasing support and participation if it is to succeed. Investigation into the science of research synthesis will increase the quality, and thus the value, of the evidence found in systematic reviews. More efficient searching methods with which to identify systematic reviews will be developed. Patients and administrators will have increasing access to reviews, which will spur practitioners to locate, appraise, and apply reviews in their practice.

Information technology can provide point-of-care access, but practitioners will need new skills to use such technology. Presentation formats will become more user-friendly for both providers and patients. Practical techniques for incorporating information from systematic reviews into clinical decision making will be created and disseminated.

While looking forward to these advances, clinicians can and should take advantage of what systematic reviews have to offer now. The Annals series will aid in that process by focusing on how to find, assess, use, and conduct systematic reviews for clinical, teaching, and research purposes. (Table 2)

Advisors: Gordon Guyatt (Hamilton, Ontario, Canada), R. Brian Haynes (Hamilton, Ontario, Canada), Andrew Oxman (Oslo, Norway), and David L. Sackett (Oxford, the United Kingdom).

Reviewer-Commentators: Iain Chalmers (Oxford, the United Kingdom), Robert Fletcher (Boston, Massachusetts), Larry V. Hedges (Chicago, Illinois), Andreas Laupacis (Ottawa, Ontario, Canada), C. David Naylor (North York, Ontario, Canada), and David L. Simel (Durham, North Carolina).

Clinical Reviewers: Clifton R. Cleaveland (Chattanooga, Tennessee), Paul F. Speckart (San Diego, California), and Norman J. Wilder (Anchorage, Alaska).

Local Editorial Assistants: Linn Morgan and Barbara Hill.

Dr. Mulrow: Audie L. Murphy Veterans Memorial Hospital, 7400 Merton Minter Boulevard, San Antonio, TX 78284.

Dr. Haynes: Health Information Research Unit, Department of Clinical Epidemiology and Biostatistics, McMaster University Medical Center, 1200 Main Street West, Hamilton, Ontario L8N 3Z5, Canada.