Researchers and clinicians rely on peer-reviewed literature to keep them informed of the latest developments in their field. Although peer review provides prestige to empirical papers, it is unclear whether or not peer review warrants this prestige. The Journal of the American Medical Association (JAMA) recently devoted a special issue to the evaluation of the peer review process and its impact on the quality of resulting publications. The articles, mostly empirical studies and scientific investigations, were compiled from presentations at the Fourth International Congress on Peer Review in Biomedical Publication held last fall. This week’s WAGER highlights some of the findings presented in the special issue and discusses the implications for gambling research.
Research Findings Presented at the Fourth International Congress on Peer Review in Biomedical Publication
Peer Review
- The editorial peer review process has not been sufficiently evaluated and its effect on the quality of publications is largely unknown (Jefferson, Alderson, Wager, & Davidoff, 2002)
Quality of Publications
- Authors often fail to include important study weaknesses in their publications (Horton, 2002)
- Results are rarely set in the context of similar research or discussed in relation to the findings of previous trials (Clarke, Alderson, & Chalmers, 2002)
- Statistical input from methodologists is inconsistently obtained and when obtained, often starts only at the analysis stage (Altman, Goodman, & Schroter, 2002)
- Few findings are expressed in the optimal manner (i.e. presenting absolute risk reduction versus relative risk reduction) (Nuovo, Melnikow, & Chang, 2002)
- Findings are extrapolated without indication beyond the tested range (Kuo, 2002)
Communication
- The journal in which a study is published is a more important determinant of how frequently the study is cited than any measure of study quality (Callaham, Wears, & Weber, 2002)
- Press releases do not accurately represent study limitations and may exaggerate the importance of the findings (Woloshin & Schwartz, 2002)
- Abstracts presented at scientific meetings are often cited by the media, yet a quarter are never published and therefore never undergo peer review (Schwartz, Woloshin, & Baczek, 2002)
Although the above studies were conducted using the general medical literature, the findings have profound implications for consumers of gambling research. For example, as we discussed in an earlier WAGER (7(20)), gambling prevalence estimates vary greatly between studies; however according to Clarke et al., authors often fail to present current research in the context of similar research. Consequently, policy makers, clinicians, scientists, and others can not rely on authors to provide this information; they must be vigilant about conducting their own checks of the literature. Similarly, it is important to recognize that many authors omit study weaknesses from media coverage and even original publications. Before implementing a new gambling treatment, clinicians would be wise to scrutinize the study for weaknesses that may reduce the efficacy, such as limitations that prevent the findings from being generalized to a larger treatment population.
Peer-reviewed literature forms the core of our knowledge about gambling and shapes everything from research initiatives to treatment practice. However, as the above list indicates, publication in a peer-reviewed journal does not guarantee high quality. In his introduction to the issue, the editor of this JAMA issue wrote “anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication” (Rennie, p. 2760). With this in mind, we urge a careful and critical reading of the literature both in peer reviewed journals and in the mass media. Only in this way will we move towards gambling policy that is based on a solid foundation of quality research.
Comments on this article can be addressed to Rachel Kidman.
References
Altman, D. G., Goodman, S. N., & Schroter, S. (2002). How statistical expertise is used in medical research. Jama., 287(21), 2817-2820.
Callaham, M., Wears, R. L., & Weber, E. (2002). Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. Jama., 287(21), 2847-2850.
Clarke, M., Alderson, P., & Chalmers, I. (2002). Discussion sections in reports of controlled trials published in general medical journals. Jama., 287(21), 2799-2801.
Horton, R. (2002). The hidden research paper. Jama., 287(21), 2775-2778.
Jefferson, T., Alderson, P., Wager, E., & Davidoff, F. (2002). Effects of editorial peer review: a systematic review. Jama., 287(21), 27842786.
Kuo, Y. H. (2002). Extrapolation of correlation between 2 variables in 4 general medical journals. Jama., 287(21), 2815-2817.
Nuovo, J., Melnikow, J., & Chang, D. (2002). Reporting number needed to treat and absolute risk reduction in randomized controlled trials. Jama., 287(21), 2813-2814.
Rennie, D. (2002). Fourth International Congress on Peer Review in Biomedical Publication. Jama., 287(21), 2759-2760.
Schwartz, L. M., Woloshin, S., & Baczek, L. (2002). Media coverage of scientific meetings: too much, too soon? Jama., 287(21), 28592863.
Woloshin, S., & Schwartz, L. M. (2002). Press releases: translating research into news. Jama., 287(21), 2856-2858.