The Quality of Drug Studies Published in Symposium Proceedings

Methods

A “symposium” is a collection of papers published as a separate issue or as a special section in a regular issue of a medical journal [4]. We defined “original clinical drug articles” as articles that 1) appeared to present original data from studies done in humans [that is, articles that had at least one table or figure that was not acknowledged to have been reprinted from another source] and 2) did not specifically state that they were reviews [4].

Selection of Articles

We identified original clinical drug articles that had a section describing the study methods, because such a section is needed to assess the quality of an article. Using a computer-generated list of random numbers from 1 to 625, we randomly selected symposia from 625 symposia that had been identified for a previous study [4]. We had data on the type of sponsorship of publication for each symposium. From each selected symposium, we randomly selected one original clinical drug article that had a methods section. We continued selecting symposia until we had enough articles (n = 127) according to the sample size estimates described below. We also calculated the proportion of articles in the selected symposia, overall and by type of sponsorship, that had methods sections.

Quality Assessment

We compared the quality of original clinical drug articles published in symposia sponsored by single drug companies with that of similar articles published in symposia that had other sponsors and in the peer-reviewed parent journals.

Sample Size Estimates

We estimated the sample size needed to test the association between the independent variable “type of sponsorship of publication” and the main outcome measure, methodologic quality score. For a three-group comparison, a minimum sample of 108 symposium articles was needed to detect a minimum effect size of 0.10 (on a scale of 0 to 1), with an α value of 0.05 and a β value of 0.80, and standard deviation of quality scores of 0.18 based on previous results [7].

To compare articles from symposia sponsored by single pharmaceutical companies with articles from the peer-reviewed parent journals, we estimated that we would need 45 symposia articles and 45 journal articles; this estimate was the result of sample size calculations done using the variables described above. Because date of publication, journal, and therapeutic class of drug could have confounded the association between source of publication and quality [8-10], we matched each symposium article to an article from the parent journal by using these characteristics, as described previously [7]. Our sample of symposium articles contained 50 articles sponsored by single drug companies, but 5 articles published in Transplantation Proceedings were excluded from this analysis because no parent journal is associated with that publication.

Instruments

We used previously developed instruments to measure the methodologic quality of articles (defined as the minimization of systematic bias and the consistency of conclusions with results) and nonmethodologic indices of quality, such as clinical relevance and generalizability. Both instruments were valid and reliable and have been published elsewhere [7].

Four reviewers independently assessed each article: Two used the methodologic quality instrument, and two used the clinical relevance instrument. We derived methodologic quality and clinical relevance scores for each article by using a previously described scoring system [7]. Each score was between 0 (lowest quality) and 1 (highest quality) and was the average of the scores of the two reviewers. Two clinical pharmacologists with extensive research experience in the health sciences did the methodologic quality assessment. For the clinical relevance instrument, three pairs of reviewers with clinical experience in general internal medicine and research experience in the health sciences each assessed one third of the articles. Each pair of reviewers reviewed the articles in the same randomized order.

For both instruments, reviewers were trained as described previously [7]. For the quality assessments, each reviewer worked independently, was blinded as to whether an article had been published in a symposium, and was given photocopies of articles from which author names, institution names, journal names, dates, and all other reference information had been obliterated. Reviewers were unaware of our hypotheses and the purpose of their reviewing, and they were paid for their work. None of the reviewers were known to us or knew of our previous work before the study.

We assessed the inter-rater reliability of quality scores by using the Kendall coefficient of concordance (W) with adjustment for tied ranks [11] and the intraclass correlation (R; treating both reviewers and articles as random effects [12]). Inter-rater reliability of quality scores was high (for methodologic quality scores: W equals 0.85, R equals 0.74 [95% CI, 0.67 to 0.80]; for clinical relevance scores: W equals 0.77, R equals 0.56 [CI, 0.44 to 0.65]).

Drug Company Support and Study Outcome

For each article, one of us determined whether a drug company had supported the research and whether the article 1) reported an outcome favorable to the drug of interest, 2) did not report an outcome favorable to the drug of interest, or 3) did not test a hypothesis. The drug of interest (as defined from the perspective of the authors, according to Gotzsche [13]) was the newest drug if two or more drugs were studied. We defined research as having had drug company support if the article that reported the research acknowledged either that a drug company had provided funding or drugs or that any of the authors were employed by a drug company. We determined drug company support solely on the basis of information in the paper.

If an article did not test a hypothesis, it was excluded from this analysis. We classified the remaining articles as “favorable” or “unfavorable” using Gotzsche's definitions [13]. An article was favorable if the drug that seemed to be of primary interest to the authors had the same effect as the comparison drug or drugs but with less pronounced side effects, had a better effect without more pronounced side effects, or was preferred more often by patients when the effect and side-effect evaluations were combined. All other articles were considered not favorable. The conclusions of the authors were taken at face value, even if they conflicted with the study results. To test inter-rater reliability, the other author independently assessed a subset of the articles (n = 90). Agreement in classifying articles as favorable or not favorable was 85%.

Statistical Analyses

Because methodologic quality and relevance scores were distributed normally (Shapiro-Wilk test), we analyzed differences between groups (type of sponsorship of publication) by using parametric one-way analysis of variance followed by the Tukey test for multiple comparisons or two-way analysis of variance (total error rate, 0.05). We compared matched groups (symposium articles and peer-reviewed parent journal articles) by using the paired t-test (two-tailed α equals 0.05).

To analyze categorical data on the outcome of studies, we tested for differences in proportions between groups by using the chi-square statistic. For tests of significance, we used an α value of 0.05. All hypothesis tests were two-sided.

Results

Presence of a Method Section

To obtain 127 original clinical drug articles for quality assessment, we had to select 213 symposia containing a total of 5041 articles. The proportions of articles that reported original data but contained no methods sections were 4% overall (195 of 5041), 10% (108 of 1064) in the symposia sponsored by single drug companies, 3% (58 of 2314) in the symposia that had other sponsors, and 2% (29 of 1663) in the symposia that had no mentioned sponsor (P < 0.001).

Quality Assessment

Quality Scores: Symposium Articles

The average methodologic quality score of all symposium articles (±SD) was 0.54 ± 0.15 (range, 0.12 to 0.91) on a scale of 0 to 1. Quality scores analyzed by type of sponsorship are shown in Table 1. Analysis of variance indicated that the methodologic quality scores differed by type of sponsorship, but the only significant difference found by multiple-comparison testing was between the methodologic quality scores of articles published in symposia sponsored by single drug companies and those of articles published in symposia with no mentioned sponsor (P < 0.05).

This small but significant difference in quality might have been due to the differential use of study designs. For example, articles in symposia sponsored by single drug companies may have had higher methodologic quality scores because they had more randomized, controlled trials, which would be expected to receive higher scores than trials with non-randomized designs.

Therefore, we examined the contribution to the variability of quality scores of study design and sponsorship by using these variables as main effects in a two-way analysis of variance and examined the interaction between these variables. Type of sponsorship was not a significant main effect (P > 0.2) study design (classified as experimental randomized, experimental unrandomized, or observational) was a significant main effect (P < 0.001). The interaction between type of sponsorship and study design was also significant (P = 0.04), indicating that the effect of study design on quality score depended on sponsorship. Table 2 shows the mean quality scores and proportion of articles by type of sponsorship and study design. These results indicate that randomized trials have higher quality scores than studies with other designs and that articles in symposia sponsored by single drug companies do include more randomized trials than do articles with other types of sponsors. These results also show that, in our sample, articles sponsored by single drug companies had the highest mean quality score for observational studies and the lowest mean quality score for randomized trials.

The mean clinical relevance score of symposium articles (n = 127) was 0.63 ± 0.16 (range, 0 to 1.0). The clinical relevance scores of symposium articles did not differ significantly by type of sponsor (one-way analysis of variance; P = 0.50) (Table 1).

Quality Scores: Symposium Articles Compared with Articles in Peer-Reviewed Parent Journals

Mean methodologic quality scores were 0.58 (CI, 0.54 to 0.62) for the articles in symposia sponsored by single drug companies and 0.57 (CI, 0.52 to 0.61) for the matched articles from the parent journals (P > 0.2). Clinical relevance scores were 0.65 (CI, 0.60 to 0.70) for the articles published in symposia sponsored by single drug companies and 0.63 (CI, 0.58 to 0.68) for the matched articles from the parent journals (P > 0.2).

Drug Company Support and Study Outcome

To determine whether drug company support of reported research was associated with study outcome, we analyzed all original drug articles (n = 172 [127 symposium articles plus 45 articles from parent journals]). We first eliminated the 20 articles that did not test a hypothesis. Of the remaining 152 articles, 40 acknowledged drug company support and 112 did not. Significantly more articles favorable to the drug of interest had received drug company support than did articles that did not acknowledge receiving such support (Table 3). When we separately analyzed articles from symposia and articles from parent journals, we found similar results (data not shown).

Discussion

We found that articles published in symposia sponsored by single drug companies were similar in quality to articles published in the peer-reviewed parent journals. Drug studies published in symposia sponsored by single drug companies were also more likely to use experimental randomized study designs than were symposia with other sponsors, if they had a methods section. However, original clinical drug articles in symposia sponsored by single drug companies were more likely to lack a methods section than were articles from symposia with other sponsors. The lack of a methods section precludes readers from critically evaluating the methods and results of a paper. Articles that acknowledged support of research by the pharmaceutical industry, in symposia or in their parent journals, were more likely to report results that favored the drug of interest than were articles that did not acknowledge pharmaceutical industry involvement.

Our results concerning randomized, controlled trials are consistent with those of Rochon and colleagues [14], who found that the methodologic quality of randomized, controlled trials published in symposia was lower than that of trials published in the parent journals. Rochon and colleagues may have been able to detect statistically significant differences because they examined only randomized, controlled trials, whereas we analyzed studies with both experimental and observational designs. Our study included too few randomized, controlled trials to allow us to conduct a similar analysis with sufficient power. In our study sample, the higher proportion of randomized, controlled trials published in the symposia sponsored by drug companies may reflect the fact that U.S. drug companies are required to submit the results of at least two such studies to the Food and Drug Administration to gain approval of a new drug [15].

The proportions of articles with statistically significant (“positive”) results and articles with nonsignificant (“negative”) results that we found in our sample of drug studies were consistent with those reported for the medical literature as a whole [16-19]. Our finding that articles acknowledging drug company support are significantly more likely to report positive results than are articles with no drug company support confirms the findings of others [5, 6] and could be due to several factors. We could determine neither the relative contributions of these factors nor the influence of other variables. However, the following factors seem most likely to explain our findings. First, drug companies may be less likely than other investigators to undertake or sponsor drug studies unless evidence already suggests that a drug is effective. Second, the study questions, study designs, or comparison groups might influence outcome [20-23]. For example, because drug manufacturers are required to compare a drug with placebo to obtain Food and Drug Administration approval for that drug, industry-sponsored studies might be more likely than non-industry-sponsored studies to use placebo controls. A drug might be more likely to appear efficacious compared with a placebo rather than an alternative treatment.

Previous studies have indicated other factors that contribute to the phenomenon of positive results being published more frequently in the medical literature than negative results (known as “publication bias” or the “file drawer” phenomenon) [16-19]. One factor is that investigators are less likely to submit articles for publication if the articles have negative results. The other is that journals are less likely to publish studies with negative results. At least one study has suggested that, for medical studies, the former factor is more important [24].

We recommend that journal editors require adequate methods sections for all articles that present original data and for review articles, so that readers can decide for themselves whether a study has appropriate rigor. Authors and editors should make an effort to publish results regardless of outcome and should be sure that funding is not tied to the ability to publish results, especially negative ones. Our results emphasize the need for the registration of clinical studies, as has been suggested [16-18, 25]. A clinical trial registry would make available the status of all clinical trials from their inception and would allow any interested parties to know whether the results of the study have been published [16].

Dr. Bero: Institute for Health Policy Studies, 1388 Sutter Street, 11th Floor, San Francisco, CA 94109.