Diseases associated with tobacco use are the leading preventable cause of death globally. Technology-based cessation programs (e.g., text message interventions) have been shown to help smokers quit. Chatbots have the potential to further improve these sorts of interventions by reducing the need for human interaction through automated responsive coaching. Before implementing chatbot-based interventions on a larger scale, it is important to assess the content of chatbot smoking cessation conversations to make sure they are providing credible information. This week, as part of our Special Series on Addiction and Technology, ASHES reviews a study by Lorien Abroms and colleagues that examined the adherence of chatbots to public health guidelines for smoking cessation.
What was the research question?
How adherent are chatbots to public health guidelines for smoking cessation?
What did the researchers do?
The researchers examined the responses of three chatbots that were developed using ChatGPT. One (S.A.R.A.H.) was created by the World Health Organization (WHO) to assist users with health-related issues. The other two chatbots (BeFreeGPT and BasicGPT) were created by the study team. BeFreeGPT was designed to specifically provide resources for people trying to quit smoking, and BasicGPT is a generic chatbot without specialized training. The study team asked all three chatbots 12 common smoking cessation questions that each used the search stem, “How do I quit smoking…” (for example, “…while pregnant”), generating a total of 36 responses. They then conducted a content analysis to assess how adherent the chatbots’ responses were to nine public health smoking cessation guidelines1.
What did they find?
Overall, chatbot responses were adherent to 57.1% (5.1 out of 9) of the smoking cessation guideline items on average. The WHO’s chatbot had the highest level of adherence (72.2%), followed by BeFreeGPT (50.0%) and BasicGPT (47.8%). The chatbots’ adherence to smoking cessation guidelines varied drastically depending on the search query that was used (see Figure). “How do I quit smoking with meds” and “How do I quit smoking while pregnant” had the highest overall adherence ratings while “How do I quit smoking with hypnosis” had the lowest. In terms of specific guidelines, most chatbot responses used clear, easily understood language and included recommendations to seek out professional help to assist with smoking cessation. However, chatbots spread misinformation (e.g., replacing cigarettes with nicotine gummies) in almost a quarter of their responses.
Figure. Displays how adherent chatbots were to different smoking cessation guidelines. Click image to enlarge.
Why do these findings matter?
These findings indicate that chatbots have potential to assist people who are looking to stop smoking. However, the chatbots’ responses to smoking cessation queries were not completely adherent to public health guidelines and there were instances where chatbots spread misinformation related to smoking cessation. This indicates that people should be cautious when using chatbots to assist with addiction treatment, particularly with smoking cessation.
Every study has limitations. What are the limitations in this study?
This study used three trained chatbots designed to provide smoking cessation guidance and might not be representative of how adherent an untrained chatbot would be to smoking cessation guidelines. In addition, the researchers coded a relatively small number of search queries, which all used the same search stem (i.e., “How do I quit smoking…”). This limits our understanding of how chatbots might respond to a larger variety of search queries or to different search stems.
For more information:
Individuals who wish to stop or cut back on their nicotine use may benefit from visiting the CDC webpage on smoking cessation. Others may benefit from visiting the American Lung Association’s quit smoking webpage. Additional resources can be found at The BASIS Addiction Resources page.
—Seth McCullock, PhD
Want CE credit for reading BASIS articles? Click here to visit our Courses Website and access our free online courses.
________________
1. Guidelines were developed to reflect recommendations from the United States Preventive Services Task Force and the National Cancer Institute, and common counseling practices. They include best practices like providing information on managing cravings and recommendations to seek professional counseling.