Compared to traditional paper and pencil methods, online studies can benefit from the convenience, lower cost, reduced time to completion, access to larger samples, and reduced self-presentation biases offered by the Internet (Miller, 1997; Schmidt, 1997). Reducing the extent that participants bias their self-presentation can benefit addiction-related research. Until now, researchers have not studied the reliability or validity of commonly used addiction-related surveys administered via the Internet. This week’s DRAM reviews a study that, in part, compares the reliability of web-based addiction-related surveys to traditional paper and pencil surveys.
Miller, E.T., Neal, D.J., Roberts, L.J., Baer, J.S., Cressler, S.O., Metrik, J., Marlatt, G.A. (2002) administered a survey to a sample of 255 undergraduates who were assigned randomly to one of three experimental conditions: pencil and paper (PP), web-based (W), and web-based with interruption (WI). The PP and W groups completed the questionnaire without stopping, but the WI group stopped for 1 to 48 hours before finishing the survey. All groups completed the same questionnaire again one week later under the same experimental conditions. In addition to basic demographic information and drinking patterns, the survey included the Alcohol Use Disorders Identification Test (AUDIT;Babor, De la Fluente, Saunders, & Grant, 1992), the Alcohol Dependence Scale (ADS: Skinner & Allen, 1982), the University of Rhode Island Change Assessment (URICA; McConnaughy, DiClemente, Prochaska, & Velicer, 1993) and the Rutgers Alcohol Problem Index (RAPI; White & Labouvie, 1989).
Figure. Means scores at time 1, correlation with score at time 2, with significant differences across condition. Click image to enlarge.
The groups did not differ on demographic variables, including age, gender, ethnicity, residence, and also did not differ on measures of alcohol use. Using correlations to measure agreement between first and second administration of the test instruments, the researchers found for all groups that all Pearson correlations were statistically significant (p < .01) and ranged from .54 to .93. Between group comparisons indicated that few differences were statistically significant out of the 48 comparisons (see Figure). The authors suggested that the indicated differences might be chance observations. We also suggest that some statistically significant differences were not clinically meaningful because they were between correlations that both indicated temporal reliability.
Limitations of the study include that the college sample might not generalize to other population segments (e.g., adults, children, etc.). Also, a one week interval might not be long enough to reduce possible memory effects. Another limitation is that the researchers dismissed some significant test-retest differences that were quite large; for example, the AUDIT dependence scale. Finally, the study utilized a self-report methodology, which could mean participants consistently presented the same inaccurate information on both tests, yielding acceptable reliability estimates at the cost of validity.
This study shows the promise of online survey techniques, and researchers might glean various benefits by using this technology. Important benefits might include: cutting research costs, reaching more and different people, and reducing time and travel restrictions on potential research participants. Researchers should continue to study this methodology to further understand its benefits and costs.
—Leslie Bosworth
What do you think? Please use the comment link below to provide feedback on this article.
References
Babor, T. F., De la Fluente, J. R., Saunders, J., & Grant, M. (1992). The alcohol use disorders identification test: Guidelines for use in primary health care. Geneva, Switzerland: World Health Organization.
McConnaughy, E. A., DiClemente, C. C., Prochaska, J. O., & Velicer, W. F. (1993). Stages of change in psychotherapy: Measurement and sample profiles. Psychotherapy: Theory, Research and Practice, 20, 368-375.
Miller, E. T. (1997). Predicting successful self-initiated health-related behavior change in the context of New Year’s resolutions: Utilizing the Internet for survey research. University of Washington.
Miller, E. T., Neal, D. J., Roberts, L. J., Baer, J. S., Cressler, S. O., Metrik, J., & Marlatt, G. A. (2002). Test-retest reliability of alcohol measures: Is there a difference between Internet-based assessment and traditional methods? Psychology of Addictive Behaviors, 16(1), 56-63.
Schmidt, W. C. (1997). World Wide Web survey research: Benefits, potential problems, and solutions Behavior Research Methods, Instruments and Computers, 29, 274-279.
Skinner, H. A., & Allen, B. A. (1982). Alcohol dependence syndrome, measurement and validation. Journal of Abnormal Psychology, 91, 199-209.
White, H. R., & Labouvie, E. W. (1989). Towards the assessment of adolescent problem drinking. Journal of Studies on Alcohol, 50, 30-37.
Catherine Mbinya July 7, 2018
Quite informative.
Anonymous February 28, 2019
yes correct don`t be a technology addict. it spoil our brain also increase the chances to get depressed harmful affect on our body posture and eyes.
couponstechie March 1, 2019
i am quitely satisfied with this it ruins our brain we have to take some actions regarding this.
eunah May 21, 2019
unfortunately most of the millennials are technology addicts and if something is not done it will only get worse
Simon Kiplangat Bett May 23, 2019
With technology, everything is becoming easy and fast
Simon Kiplangat Bett May 24, 2019
The trend is there for everyone to see. we are now dependent on technology and it is going to get worse
chrispinus Kulenya May 29, 2019
I like the fact that reducing the extent that participants bias their self-presentation can benefit addiction-related research. Very informative
Keep up
Stephen Kiboi June 20, 2019
This is an eye opener.
jenny January 23, 2020
Great article. Couldn’t be written much better!