Can the Open Science Revolution Revolutionise Gambling Research?

Editor’s Note: This editorial was written by Drs. Robert Heirene & Sally Gainsbury as part of our Special Series on Open Science Practices.

The crisis

Over the last 8-10 years there has been a sharp increase in meta-research––that is, research of research; studies of studies. These studies have highlighted a host of poor practices used by scientists when conducting and reporting their research, including P-hacking, (S)HARKing, biased publication decisions, selective outcome reporting, and the use of small, under-powered sample sizes (see Box 1). These practices pose serious concerns for our confidence in the scientific enterprise and are thought to be largely responsible for the inability to replicate many of the findings obtained in earlier, seminal studies––particularly in psychology. It comes as no surprise, therefore, that the resulting narrative adopted by the media and academics alike has been sensationalist, including ever increasing uses of the term “crisis” and calls for “revolutions.”

Box 1: Poor research practices plaguing psychological research

P-hacking: deliberately searching for and/or manipulating data to achieve results that are statistically significant.

(Secretly) Hypothesising After the Results are Known or (S)HARKing: presenting unanticipated findings as if they were predicted before the start of a study.

Biased publication decisions: only publishing studies in which hypotheses are confirmed or in which statistically significant results are achieved. This relates to wider issues in the scientific community of favouring statistically significant and/or novel results over null findings and replications, resulting in an overall publication bias.

Selective outcome reporting: selectively choosing outcomes to report, typically based on whether they are statistically significant or not.

Under-powered samples: the inclusion of small sample sizes in studies, resulting in a lack of statistical power or the ability to identify a true effect when it exists (false-negative) or, conversely, leading to finding an effect that does not exist (false-positive).

 

Gambling research faces several additional problems or crises that are unique to the area, including accusations of bias within work that involves in-kind or direct support from gambling industry or local governments; related concerns that the source of funding will influence research questions, designs, or reporting; bias in terms of anti-gambling moralistic ideology; and resistance to publishing research that demonstrates any benefits of gambling. Moreover, many of the theories and principles we use to understand gambling behaviour and conceptualisations of problem gambling were developed from research conducted using now-outdated slot machines and older populations. This no longer reflects the reality of gambling, which has enthusiastically embraced new technologies and transformations including virtual and augmented reality gambling, blockchain and cryptocurrency, gambling within online games, and skill elements of gaming machines. Younger generations are gambling in very different ways as mobile and Internet gambling rapidly increases internationally.

The revolution

Concerns over bias and low replication rates within scientific research have prompted a transition towards more rigorous and transparent ways of doing research known as the open science movement. Proponents of this movement advocate for practices such as pre-registering one’s research questions, hypotheses, methodology, and data analysis plans in a time-stamped document stored in an accessible online repository; adopting strategies for transparently and accurately reporting outcomes; openly sharing raw data collected and materials used; and actively pursuing the replication of older studies. Recently, the editors of the journal International Gambling Studies (including one of the present authors [SG]) announced that the journal will now encourage the adoption of these practices and offer Open Science Badges for authors who pre-register their study and those who share their raw data and study materials. In the same issue of the journal, two additional essays were published by leading gambling researchers which discussed the need for replications of gambling studies and the adoption of open science within the field. Here, we contribute to this discussion by further outlining how the adoption of an open science approach to conducting gambling research can be used to prevent poor research practices and tackle the problems unique to gambling studies, including questions regarding the ability of scientists to conduct unbiased, credible research under the auspices of relevant stakeholders (e.g., industry, industry-associated affiliates, local governments) and the urgent need to modernise gambling research.

Removing concerns surrounding stakeholder-backed research

Concerns relating to the credibility of stakeholder-backed research are primarily focused on the influence that stakeholders may have over the research questions asked, the design of studies, or decisions regarding whether to publish and which outcomes to report[1]. These latter concerns could be addressed through the publication of a comprehensive pre-registered study protocol that is developed independent of stakeholders. By clearly stating study hypotheses and data analysis plans, researchers can allay concerns that stakeholders have meddled in the analysis process or selectively chosen the outcomes to be reported. Pre-registration can also preclude any pro-industry biases held by researchers (conscious or unconscious) from affecting ad hoc decisions made during analysis and reporting by restricting researcher degrees of freedom[2]. Ideally, pre-registrations should be completed prior to involving funding bodies, thereby removing questions as to whether they contributed to research questions or design, although this may be difficult to achieve in practice. Transparent reporting can increase confidence in stakeholder-funded research by ensuring all outcomes studied are reported, regardless of whether they align with perceived industry interests or not. In gambling research, transparent reporting should include clear statements outlining the source(s) of funding and any roles played by stakeholders in the research process. Finally, by sharing the data collected in these studies, researchers enable others to verify their findings and ensure all analyses were undertaken properly and without bias.

Modernising gambling research

Much of the evidence that underpins conceptualisations of gambling involvement and gambling problems is potentially outdated. This issue is directly influenced by publication bias––replication studies, especially those challenging the status quo, have historically not been prioritised in gambling research (or psychological research more broadly), meaning the results of older studies have stood unchallenged in the absence of modern replications. Open science practices such as pre-registering can discourage decisions not to publish on the basis of null findings, while using the registered report format[3] can remove this concern altogether. By obviating the need to produce statistically significant findings, researchers can concentrate on conducting methodologically rigorous gambling studies and are less inclined (and able) to engage in P-hacking or selective outcome reporting. However, a change in scientific values is required on the behalf of journals, editors, and reviewers in the gambling field so that replications and null findings are not disincentivised, leaving those engaging in open science disadvantaged. There is an onus on these parties to challenge any reliance on citations of older studies to support claims about how gamblers and industry operate in the technology-laden 21^st century. Similarly, policy makers and gambling regulators should base policies and practices on recent evidence and support research as necessary to enable this.

Progress so far

Pre-registration: There is good reason to believe that the pre-registration of gambling studies is increasing. A quick search for registrations with “gambling” in the title on the Open Science Framework returns a small but promising number (59), most of which were registered in 2019 or 2018. However, a deeper look at some of these pre-registrations reveals poor practices that would be likely to engender further mistrust if the research were industry-funded, including scantily described randomization procedures, absent justifications for seemingly arbitrary sample sizes, and poorly outlined analysis plans. Systematic review is necessary to confirm these observations; however, at this point, we believe we must do better if we want to increase confidence in the outcomes of stakeholder-backed gambling research. Decisions will be needed as to who has responsibility to peer-review preregistrations as this creates additional burdens for journal editors, reviewers and the academic community which may slow down the research process. Open science platforms could require (and check) that authors report on a minimal set of criteria before their preregistration can be published. Authors could pay an additional fee to journals (similar to open access fees) to enable checks and ‘badges’ to accompany their articles, or reviewers could be incentivised in some manner to do this. Journals should encourage authors to pre-register studies by asking them to provide a statement of where the study is pre-registered or justifying why it was not. Academic institutions should train researchers in the importance of these practices and encourage this through recording these as metrics for promotion.

Transparent reporting: Progress in this domain is more difficult to assess, though the increase in pre-registration and the general trend towards open science in all psychological research is encouraging for gambling studies. The growing number of journals that require submitting authors to adhere to reporting guidelines (e.g., PRISMA; CONSORT) is likely to result in improvements in this domain.

Data and materials sharing: Progress here is evident. The Division on Addiction have set up The Transparency Project, an online repository for sharing privately funded data sets obtained by addictions researchers. At present, the number of projects for which data are presented is modest, at 10, but all relate to gambling research. This practice is likely to increase as journals are increasingly asking authors to indicate where their materials and data are available online, or to provide a statement explaining why these could not be shared. To further improve data transparency, researchers must stipulate in contractual agreements that any data collected during privately funded projects must be able to be made publicly available in its de-identified form, and include similar declarations on ethics applications. Many government-funded research organisations internationally are now adopting data sharing as best practice and making funding reliant on open science practices is likely to encourage researchers to adopt these. 

Final comments

The principles and practices of open science are being increasingly adopted by psychological scientists as a means of improving the quality and credibility of their research. In the gambling field, adopting an open science-style approach to conducting research may be particularly important and impactful when considering the specific challenges faced by gambling researchers. Although some progress has been made, there remains considerable room for improvement. All parties involved in this enterprise, including journal editors, grant providers, research institutions, and researchers, should encourage and incentivise replication studies and transparent research practices, including pre-registration and open data sharing. 

Author biographies

Dr Robert Heirene is a Postdoctoral Research Associate at the Gambling Treatment and Research Centre at the University of Sydney, Australia. Dr Sally Gainsbury is an Associate Professor at the University of Sydney and the co-director of the Gambling Treatment and Research Centre.

Funding sources

Robert Heirene receives funding from Responsible Wagering Australia (awarded to Sally Gainsbury) for his research on the use and effectiveness of consumer protection tools by online wagering customers in Australia.

Sally Gainsbury is the recipient of an Australian Research Council Discovery Early Career Research Award [DE1060100459] and has received funding from many sources for her research, which are detailed here.

 

[1] Although, a recent review of gambling publications found no association between funding sources and eight different study characteristics, including the use of comparison groups, participant eligibility criteria used etc. (Ladouceur, Shaffer, Blaszczynski, & Shaffer, 2019)

[2] The same too could be said for researchers biased in the other direction, opposing gambling on moralistic grounds.

[3] Registered reports are peer-reviewed by a journal editor and review panel entirely based on the introduction and methods section, prior to data collection and outcomes being known.