
==== Front
9214582
21248
Stud Health Technol Inform
Stud Health Technol Inform
Studies in health technology and informatics
0926-9630
1879-8365

35673078
10.3233/SHTI220139
nihpa2024102
Article
Latent Linguistic Motifs in Social Media Postings Resisting COVID-19 Misinformation
Singh Tavleen MS
Olivares Sofia BS
Myneni Sahiti PhD
School of Biomedical Informatics, The University of Texas Health Science Center, Houston, Texas, USA
Corresponding author: Tavleen Singh, tavleen.kaur.ranjit.singh@uth.tmc.edu
20 9 2024
06 6 2022
24 9 2024
290 557561
https://creativecommons.org/licenses/by-nc/4.0/ This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0).
Social media has become a predominant source of information for many health care consumers. However, false and misleading information is a pervasive problem in this context. Specifically, health-related misinformation has become a significant public health challenge, impeding the effectiveness of public health awareness campaigns and resulting in suboptimal responsiveness to the communication of legitimate risk-related information. Little is known about the mechanisms driving the seeding and spreading of such information. In this paper, we specifically examine COVID-19 tweets which attempt to correct misinformation. We employ a mixed-methods approach comprising qualitative coding, deep learning classification, and computerized text analysis to understand the manifestation of speech acts and other linguistic variables. Results indicate significant differences in linguistic variables (e.g., positive emotion, tone, authenticity) of corrective tweets and their dissemination level. Our deep learning classifier has a macro average performance of 0.82. Implications for effective and persuasive misinformation correction efforts are discussed.

Deep learning
COVID-19
Misinformation
==== Body
pmcIntroduction

The World Health Organization (WHO) defined the COVID-19 virus (initially known as 2019-nCoV) outbreak as a severe global threat [1]. Social media has become predominant as a source of information for many health care consumers during these unprecedented times. In emerging situations and stressful circumstances such as COVID-19, where in-person interactions are discouraged and prohibited, online information dissemination provides unique opportunities and unexpected challenges for public health agencies. Specifically, viral outbreaks of false and misleading information have been a pervasive problem in this context [2, 3]. With the current climate of distrust in scientific institutions and medical systems as a source of reliable information [4], this presents a significant public health challenge, as inaccurate information may lead to harmful health behaviors and impede the effectiveness of population health interventions and responsiveness to the communication of legitimate risk-related information [5].

While misinformation in social media is not a new emergence, we have a limited understanding of how individuals respond to misinformation in various ways, such as seeding and spreading it, rejecting it, or providing further information to support or refute it. Several studies have focused on developing deep learning and computational linguistic models to identify and mitigate misinformation spread and model stance-taking behaviors of individuals in social media [6, 7]. However, these methods ignore the social processes and cognitive factors fueling the seeding and spreading of misinformation. Without understanding these important components that drive human-misinformation interaction, computational models cannot be fully optimized to reach their full potential [8, 9]. The processes of a machine-based system may differ from the conscious assessments of individuals when considering such information. Human misinformation detection and mitigation tightly couples social and cognitive processes [10, 11]. It is the intersection of the strength of reasoning, the fragility of memory [12], and interpersonal factors such as reputation. To understand how (mis)information is assessed, we need to consider the social dynamics of endorsement, authenticity, reputation, and other interpersonal considerations along with human factors in memory, attention, and reasoning [13, 14]. In order to understand effective correction strategies that can be adopted by health consumers and public health authorities, it is important to examine cognitive models of naturalistic decision making as individuals interact with misinformation and observe the enacted fate of such correction strategies in the context of the inter- and intra-personal factors affecting its dissemination in online social settings, thus, providing insights into learning pathways to increase individual resistance to misinformation that address the observed cognitive, social, and behavioral susceptibilities of individuals and communities. This will permit the characterization of misinformation intention-perception dynamics in digital social settings, ultimately allowing us to develop resilient information dissemination approaches to mitigate misinformation seeding and spread, as well as develop targeted learning strategies to help individuals spot misinformation and prevent its spread.

In this paper, we will present a novel methodological framework to examine the ways in which individuals resist and correct misinformation in online social media. To this end, we (a) conduct computerized text analysis of linguistic attributes of social media posts, (b) develop a semi-automated deep learning linguistic modeling for high throughput classification of intent using speech acts theory, and (c) model the relationships between linguistic features and the dissemination of correction posts. Such analysis forms the foundational step towards the characterization of human-information intention-perception dynamics in digital social settings, ultimately allowing us to develop scalable and reliable computational infrastructure that can help formulate resilient information dissemination approaches to negotiate and compensate misinformation perception and spread, easing public health burden and informing policy regulations as needed. Results from this research will help us formulate responsive interventions that empower individuals in effective stance-taking strategies and prevent misinformation spread in online social media.

Methods

Qualitative analysis:

Tweet data ranging from January 2020 to January 2021 was retrieved from a COVID-19 Tweet-ID repository created by Chen et al. [15]. A dataset consisting of 423,652 tweets was obtained by hydrating the provided Tweet-IDs using the Twarc package [16]. A subset of 1,400 randomly chosen tweets was created for qualitative analysis. In order to understand the ways in which COVID-19 (mis)information is expressed, we analyzed tweets using a modified version of Searle’s speech acts theory [17, 18] consisting of the following labels: declaratives (announces objective information), expressive (expresses speaker’s psychological reactions to events), desire (pursued action or result), question (information request), commissive (promise to perform an action), directive (requests an action from message recipient), emotion (expresses feelings towards a situation), assertion (states personal believes definitively), stance (conveys a standpoint), and statement (explanation). A subset of 100 tweets was independently coded by two researchers and inter-rater reliability (Krippendorff’s alpha) between the raters was calculated. Additional details about the qualitative labeling procedure and dataset characteristics can be found here [19].

Deep learning-based intent classification:

Transformer-based models like BERT have emerged as the state-of-the-art models in many NLP-related tasks primarily because of their ability to capture bidirectional contextual information [20, 21]. BERT-base model consists of 12 layers of transformer blocks, 12 attention heads (768 hidden size), and 110 million parameters [20]. For this study, we used a variant of this model called BERTweet [22], which uses the same architecture as BERT-base to perform intent classification of COVID19 misinformation tweets. BERTweet is pre-trained using a corpus consisting of 850M English Tweets, out of which 845M Tweets were obtained from 01/2012 to 08/2019 and 5M tweets were related to the COVID-19 pandemic [22].

Using the manually coded dataset, we first performed text preprocessing in order to convert the text to lowercase and also remove any hyperlinks from the textual data. We then split the entire dataset into 90%, 5%, and 5% for training, validation, and test sets, respectively. We employed a fine-tuning layer that consisted of two fully connected dense layers (768 and 512 units, respectively) and a sigmoid activation function in the output layer (5 units). We used a learning rate of 1 × 10−5. We also computed class weights for the loss function to assign a higher weight to the loss encountered by the tweets associated with minor classes. The model was trained for 20 epochs. We converted the probabilities into actual classes based on the threshold value calculated using the validation set. To evaluate the performance of the classifier, we used the held-out test set. We used the following evaluation metrics to evaluate the classifier’s predictions on the held-out test dataset: Recall – It is the number of true positives divided by the number of true positives plus the number of false negatives.

Precision – It is the number of true positives divided by the number of true positives plus the number of false positives.

F1-score - It is defined as the harmonic mean of the precision and recall.

Analysis of attempts to correct misinformation in social media:

The COVID-19 Twitter misinformation dataset called CMU-MisCov19 [23] was further analyzed in this study. This dataset was created to identify and characterize COVID-19 misinformation communities [23]. This dataset was also hydrated using Twitter’s API and the Twarc package [16]. This dataset consisted of 4573 Twitter-IDs annotated for 17 categories, including tweets calling out or correcting misinformation [23]. Of the 4573 Twitter-IDs, only 3702 tweets were available for retrieval at the time of hydration. From that subset of tweets, a total of 1204 posts were considered to be calling out or correcting COVID-19 misinformation as described in [23].

For the CMU-MisCov19 dataset, we applied our deep learning classifier described above to assign speech acts-based intent labels. Further, dissemination levels were assigned based on tweet-level metrics capturing users’ interactions with the tweets, in this case, retweets and favorites. Due to the non-normal distribution of the data, the interquartile range (IQR) of the sum of these metrics was utilized to determine the thresholds between low (0 interactions), medium (1–4 interactions), and high (>4 interactions) dissemination levels. To understand how users communicate through the use of various words as embedded in their language, we utilized Pennebaker’s Linguistic Inquiry and Word Count (LIWC) 2015 analysis [24] software to extract linguistic features of corrective COVID-19 misinformation tweets and compare them across the three dissemination levels. We also compared the linguistic features of corrective tweets with those of non-corrective tweets. These tweets were first pre-processed to improve the performance of the software by removing unrecognizable components such as hashtags, mentions, embedded web addresses, and symbols (emojis). Due to the nature of the data, the nonparametric Kruskal-Wallis test was utilized to evaluate the differences in linguistic patterns among correction tweets at various dissemination levels.

Results and Discussion

Qualitative analysis:

The distribution of different speech act classes in the manually annotated dataset is shown in Figure 1. The most prevalent speech act was assertion (n=445), followed by declaratives (n=373), statement (n=303), directive (n=300), question (n=204), expressive (127), emotion (n=100), stance (n=96), desire (n=73), and commissive (n=42). The inter-rater reliability between the two coders was 0.84 for labeling speech acts. Given the imbalanced distribution of speech act classes in our manually coded dataset, we built the multilabel intent classifier for the five most prevalent speech act classes consisting of 1289 manually coded tweets.

Table 1 provides illustrative examples of tweets and manifestations of various speech acts categories. Users expressed different speech acts as per their Twitter conversations. Assertion speech acts focused more on individual’s beliefs, whereas commissive highlighted the seriousness some users exhibited while dealing with the pandemic situation. Expressive and emotion speech acts reflected individuals’ state of mind towards the spread of the pandemic. Twitter users also used the platform to request information about coronavirus in the form of questions or queries. Users also announced relevant information on Twitter in the form of declaratives.

Deep learning-based intent classification:

The overall micro and macro average of the classifier was high – 0.81 and 0.82 respectively (Table 2). In terms of specific speech acts classes, declaratives speech act had the highest F1 score of 0.92, followed by question speech act which had a F1 score of 0.88, and directive speech act which had an F1 score of 0.82. The performance for the assertion speech act was reasonable, with an F1 score of 0.77. The performance for the statement speech act was low, with an F1 score of 0.69. The post-hoc error analysis using the test set predictions revealed that the lowest performance for statement speech act class could be attributable to – (a) statement class was mostly missed when it co-occurred with other classes, (b) embedded links which were used during qualitative analysis was irrelevant to improve classifier performance.

Figure 2 shows the distribution of speech acts within the misinformation correction tweets across three dissemination levels. Declaratives and question speech acts were more prevalent in the low dissemination tweets (n=176, n=77 respectively) as compared to the high (n=149, n=44 respectively) or medium (n=128, n=44 respectively) dissemination tweets. The expression of assertion and directive speech acts was comparable across medium (n=219, n=87 respectively) and low (n=229, n=86 respectively) dissemination tweets. Question speech act was equally distributed across the high and medium dissemination tweets (n=44 each). Statement speech act was most prevalent in the medium dissemination tweets (n=39) as compared to high (n=28) or low (n=34) dissemination tweets.

Analysis of attempts to correct misinformation in social media:

Table 3 shows the LIWC categories that had a significant association with dissemination levels. LIWC analysis showed that the mean word count was the highest in high dissemination tweets as compared to medium or low dissemination tweets, which indicates that the tweets that contained more thorough explanations had higher influence as compared to other tweets. This difference was also statistically significant (<0.05). The words containing more than six letters (SixItr) had higher usage in low dissemination tweets than high or medium dissemination tweets which reflects that the more convoluted messages are, the more difficult they may be to understand. Additionally, the expression of more positive emotions in the high dissemination tweets reflects the emphasis on perceived benefits, which may be a more successful correction strategy. The use of more inclusive personal/inclusive words (I, we, friend, drives, affiliation) in high dissemination correction tweets may indicate that using personal/inclusive information may resonate with others (via emotional proximity), making it a successful strategy.

Table 4 shows the LIWC categories that had a significant association to corrective (n=1204) vs. non-corrective tweets (n=2498). In the corrective tweets, the less expression of LIWC word categories such as analytic, reward, and compare may reflect areas of improvement (increasing objectivity or confidence or assertiveness or emphasizing benefits or elaborate more on reasoning-why is it better to change behavior?) when addressing/correcting misinformation. In terms of LIWC categories such as dictionary words and authenticity, corrective information may be more articulate and trustworthy, whereas non-corrective tweets may have been impacted by misinformation. Higher mean values for tone, negative emotion, and anger in corrective tweets may reflect frustration provoked by misinformation dissemination. The differences in LIWC categories such as negate, insights, and differ may reflect differences in expression of corrective measures focusing on contradicting misinformation (negate, differ) and emphasizing risks and insights (think, know) about present events/actions (present focus). LIWC category of social may reflect differences in expression of implications or effects of actions or events, indicating that corrective information emphasizes community-level trends and impacts.

Our work is not without limitations. Since the distribution of speech acts was not balanced in our manually coded dataset, we included only the top five most prevalent speech act classes in fine-tuning the deep learning model. This led to the omission of other speech act classes during the classification of the corrective misinformation Twitter dataset. It is essential to look at all the speech act classes within the corrective misinformation dataset to have a comprehensive understanding of how individuals take a stance against misinformation. In our future work, we will focus on increasing our manual coding efforts and adding additional features in the language models to improve the performance of the intent classification task. Given the retrospective organizational review policies [25], several tweets were not retrieved when the dataset was hydrated (e.g., deleted tweets, account suspensions). However, the subset of tweets analyzed in this study were selected at random, mitigating the impact of these limitations. Other challenges that limit the use of Twitter for data collection include restrictions on the number of requests that can be made to the Twitter API, limited representativeness of the internet users, fictitious accounts, costs involved in obtaining Twitter data, privacy and ethical issues, etc. [26].

Conclusions

Health misinformation circulates in social media and continues to affect the ways in which individuals perceive the health consequences of these behaviors, even though several of these misleading claims have been refuted by global, national, and local public health agencies and scientific associations. However, inter- and intra-individual processes that facilitate the correction or spread of such misleading information are not well understood. Our work leverages qualitative coding and deep learning frameworks, coupled with linguistic inquiry, to describe cognitive mechanisms and social processes that inform and shape individuals’ efforts to resist misinformation spread and to model human-misinformation interaction dynamics. Significant differences in linguistic variances between dissemination levels of corrective tweets allow us to identify active ingredients that are imperative to deliver and disseminate persuasive health messaging. Ultimately, such understanding will allow us to formulate responsive educational interventions that enable individuals to identify and correct misinformation and prevent its spread in online social media.

Acknowledgements

This research was partly supported by the National Library of Medicine of the National Institutes of Health under award numbers 1R01LM012974-01A1 and 3R01LM012974-02S1. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Figure 1– Distribution of speech acts (n=1400)

Figure 2– Distribution of speech acts within misinformation correction tweets

Table 1- Definitions of speech acts and example tweets

Speech acts	Definition	Example	
Assertion	States the speaker’s own beliefs to attempt to persuade or obtain hearer’s engagement	With the rise of the 2019-nCoV strain of coronavirus, I figured a rewatch of Contagion (2011) was in order.	
Commissive	Commits the speaker to perform an action	We need to take things seriously, but keep the heid and follow the advice of the medical professionals.	
Declaratives	Announces information which might result in a change of affairs or reach a decision	CDC confirms first US case of coronavirus that has killed 9 in China	
Desire	Sought action or result	I’ve been hearing this! I hope it’s true. When I heard there was an HIV component to Coronavirus, my heart dropped.	
Directive	Efforts by the speaker to make the hearer perform an action	‘No reason for Americans to panic’: White House looks to calm fears over coronavirus	
Emotion	Expresses personal reactions and feelings towards a circumstance	I don’t like that song even when we aren’t in a pandemic. ??	
Expressive	Expresses speaker’s psychological condition about present conditions or themselves	My work assumed I had the corona virus because my boyfriend is Chinese when I was sick from my chronic illness??	
Question	Requests information	#Coronavirus: Where does it come from and how did it mutate to infect humans?	
Stance	Portrays a standpoint	Great step taken by “ID12XX” we will takeover this corona soon and get back to normal life with more power.	
Statement	Descriptive, expositive	My Virus, called Coronavirus, just wiped out the world in 579 days!	

Table 2- Performance evaluation of BERTweet model for speech acts classification

Per Class Performance	
Speech acts	Precision	Recall	F1	
Assertion	0.67	0.91	0.77	
Declaratives	0.90	0.95	0.92	
Directive	0.75	0.90	0.82	
Statement	0.77	0.62	0.69	
Question	1.00	0.79	0.88	
Overall Model Performance	
Micro avg.	0.79	0.84	0.81	
Macro avg.	0.82	0.83	0.82	
Weighted avg.	0.81	0.84	0.81	

Table 3- Mean (SD) corrective tweets LIWC word counts among dissemination levels

Category	High	Medium	Low	H	p	
tone	34.3 (35.7)	26.8 (31.5)	29.5 (33.5)	6.6	0.036	
words/sent ence	29.5 (12.1)	28.7 (12.8)	25.2 (12.4)	28.6	<0.001	
words>6 letters	25.8 (11.1)	24.8 (12.2)	27.7 (12.5)	13.4	0.001	
dictionary words	75.9 (12.4)	75.5 (13.7)	72.0 (15.1)	16.7	<0.001	
i	1.8 (3.8)	1.8 (3.4)	1.0 (2.6)	22.6	<0.001	
we	0.5 (1.6)	0.5 (1.7)	0.4 (1.4)	8.8	0.013	
positive emotions	2.4 (3.3)	1.9 (2.8)	2.2 (4.0)	6.0	0.049	
friend	0.4 (1.3)	0.2 (1.0)	0.1 (0.8)	15.5	<0.001	
drives	7.6 (5.8)	6.4 (5.3)	7.1 (5.6)	7.5	0.024	
affliation	2.1 (3.1)	1.6 (3.0)	1.6 (3.0)	10.1	0.006	
leisure	1.2 (2.5)	0.9 (2.1)	1.1 (2.6)	6.1	0.046	

Table 4- Mean (SD) corrective vs. non-corrective tweets LIWC word counts

Category	Corrective	Non-corrective	H	p	
tone	30.1 (32.7)	36.5 (35)	43.9	<0.001	
negative emotion	3.7 (4.7)	2.6 (3.8)	62.4	<0.001	
anger	1.6 (3.2)	1.1 (2.5)	19.8	<0.001	
dictionary words	74.4 (14)	70.9 (14)	62.7	<0.001	
authentic	26.9 (29.7)	24.3 (29.8)	19.7	<0.001	
analytic	60.7 (32.6)	72.4 (29.0)	118.7	<0.001	
reward	1 (2.1)	1.2 (2.5)	6.0	<0.001	
compare	1.6 (2.9)	2.0 (3.4)	5.6	<0.001	
negate	3.6 (4.5)	1.6 (3.3)	250.2	<0.001	
differ	4.3 (4.7)	2.6 (3.7)	122.8	<0.001	
insights	2.4 (3.4)	1.8 (3.0)	42.6	<0.001	
social	8.7 (6.9)	7.6 (6.7)	23	<0.001
==== Refs
References

[1] World Health Organization (2020). Coronavirus disease 2019 (COVID-19): situation report, 73. World Health Organization. https://apps.who.int/iris/handle/10665/331686
[2] Singh L , Bansal S , Bode L , Budak C , Chi G , Kawintiranon K , Padden C , Vanarsdall R , Vraga E , and Wang Y , A first look at COVID-19 information and misinformation sharing on Twitter, ArXiv. (2020) arXiv:2003.13907v1.
[3] Mian A , and Khan S , Coronavirus: the spread of misinformation, BMC Med. 18 (2020) 89. doi:10.1186/s12916-020-01556-3.32188445
[4] Vosoughi S , Roy D , and Aral S , The spread of true and false news online, Science. 359 (2018) 1146–1151. doi:10.1126/science.aap9559.29590045
[5] Chou W-YS , Oh A , and Klein WMP , Addressing Health-Related Misinformation on Social Media, JAMA. 320 (2018) 2417–2418. doi:10.1001/jama.2018.16865.30428002
[6] Bal R , Sinha S , Dutta S , Joshi R , Ghosh S , and Dutt R , Analysing the Extent of Misinformation in Cancer Related Tweets, Proceedings of the International AAAI Conference on Web and Social Media. 14 (2020) 924–928.
[7] Hou R , Perez-Rosas V , Loeb S , and Mihalcea R , Towards Automatic Detection of Misinformation in Online Medical Videos, in: 2019 International Conference on Multimodal Interaction, Association for Computing Machinery, New York, NY, USA, 2019: pp. 235–243. doi:10.1145/3340555.3353763.
[8] Fell L , Gibson A , Bruza P , and Hoyte P , Human Information Interaction and the Cognitive Predicting Theory of Trust, in: Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, Association for Computing Machinery, New York, NY, USA, 2020: pp. 145–152. doi:10.1145/3343413.3377981.
[9] Lewandowsky S , Ecker UKH , and Cook J , Beyond Misinformation: Understanding and Coping with the “Post-Truth” Era, Journal of Applied Research in Memory and Cognition. 6 (2017) 353–369. doi:10.1016/j.jarmac.2017.07.008.
[10] Metzger MJ , Flanagin AJ , and Medders RB , Social and Heuristic Approaches to Credibility Evaluation Online, Journal of Communication. 60 (2010) 413–439. doi:10.1111/j.1460-2466.2010.01488.x.
[11] Gasser U , Cortesi SC , Malik M , and Lee A , Youth and Digital Media: From Credibility to Information Quality, SSRN Journal. (2012). doi:10.2139/ssrn.2005272.
[12] Cook J , Ecker U , and Lewandowsky S , Misinformation and How to Correct It, in: Emerging Trends in the Social and Behavioral Sciences, American Cancer Society, 2015: pp. 1–17. doi:10.1002/9781118900772.etrds0222.
[13] Swire-Thompson B , and Lazer D , Public Health and Online Misinformation: Challenges and Recommendations, Annu Rev Public Health. 41 (2020) 433–451. doi:10.1146/annurevpublhealth-040119-094127.31874069
[14] Ecker UKH , Hogan JL , and Lewandowsky S , Reminders and Repetition of Misinformation: Helping or Hindering Its Retraction?, Journal of Applied Research in Memory and Cognition. 6 (2017) 185–192. doi:10.1016/j.jarmac.2017.01.014.
[15] Chen E , Lerman K , and Ferrara E , Tracking Social Media Discourse About the COVID-19 Pandemic: Development of a Public Coronavirus Twitter Data Set, JMIR Public Health and Surveillance. 6 (2020) e19273. doi:10.2196/19273.32427106
[16] Summers E . DocNow/twarc. GitHub Web site. https://github.com/DocNow/twarc. Updated 2020. Accessed Jan 6, 2021.
[17] Searle JR , Speech Acts: An Essay in the Philosophy of Language, Cambridge Core. (1969). doi:10.1017/CBO9781139173438.
[18] Austin JL , How To Do Things With words: The William James Lectures Delivered at Harvard University in 1955, Oxford University Press, Oxford, New York, 1975.
[19] Olivares SM , and Myneni S . Unpacking misinfodemic during a global health crisis: a qualitative inquiry of psychosocial characteristics in social media interactions. [Accepted] MedInfo 2021.
[20] Devlin J , Chang M-W , Lee K , and Toutanova K , BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, ArXiv:1810.04805 [Cs]. (2019). http://arxiv.org/abs/1810.04805 (accessed March 30, 2021).
[21] Vaswani A , Shazeer N , Parmar N , Uszkoreit J , Jones L , Gomez AN , Kaiser L , and Polosukhin I , Attention Is All You Need, ArXiv:1706.03762 [Cs]. (2017). http://arxiv.org/abs/1706.03762.
[22] Nguyen DQ , Vu T , and Nguyen AT , BERTweet: A pre-trained language model for English Tweets, ArXiv:2005.10200 [Cs]. (2020). http://arxiv.org/abs/2005.10200.
[23] Memon SA , and Carley KM , Characterizing COVID-19 Misinformation Communities Using a Novel Twitter Dataset, ArXiv:2008.00791 [Cs]. (2020). http://arxiv.org/abs/2008.00791.
[24] Pennebaker J , Francis M , and Booth R , Linguistic inquiry and word count (LIWC), Erlbaum Publishers, Mahwah, NJ, 2001.
[25] Developer Policy – Twitter Developers, (2021). https://developer.twitter.com/en/developer-terms/policy (accessed September 7, 2021).
[26] Zhang AJ , Albrecht L , and Scott SD , Using Twitter for Data Collection With Health-Care Consumers: A Scoping Review, International Journal of Qualitative Methods. 17 (2018) 1609406917750782. doi:10.1177/1609406917750782.
