Australasian Institute of Digital Health Summit 2022–Automated Social Media Surveillance for Detection of Vaccine Safety Signals: A Validation Study

Abstract

Background Social media platforms have emerged as a valuable data source for public health research and surveillance. Monitoring of social media and user-generated data on the Web enables timely and inexpensive collection of information, overcoming time lag and cost of traditional health reporting systems.

Objectives This article identifies personally experienced coronavirus disease 2019 (COVID-19) vaccine reactions expressed on Twitter and validate the findings against an established vaccine reactions reporting system.

Methods We collected around 3 million tweets from 1.4 million users between February 1, 2021, to January 31, 2022, using COVID-19 vaccines and vaccine reactions keyword lists. We performed topic modeling on a sample of the data and applied a modified F1 scoring technique to identify a topic that best differentiated vaccine-related personal health mentions. We then manually annotated 4,000 of the records from this topic, which were used to train a transformer-based classifier to identify likely personally experienced vaccine reactions. Applying the trained classifier to the entire data set allowed us to select records we could use to quantify potential vaccine side effects. Adverse events following immunization (AEFI) referred to in these records were compared with those reported to the state of Victoria's spontaneous vaccine safety surveillance system, SAEFVIC (Surveillance of Adverse Events Following Vaccination In the Community).

Results The most frequently mentioned potential vaccine reactions generally aligned with SAEFVIC data. Notable exceptions were increased Twitter reporting of bleeding-related AEFI and allergic reactions, and more frequent SAEFVIC reporting of cardiac AEFI.

Conclusion Social media conversations are a potentially valuable supplementary data source for detecting vaccine adverse event mentions. Monitoring of online observations about new vaccine-related personal health experiences has the capacity to provide early warnings about emerging vaccine safety issues.

Keywords

COVID-19 vaccines - social media - Internet - artificial intelligence - patient-generated health data - pharmacovigilance - immunization - vaccination - adverse events

Background and Significance

Vaccine safety is a critical component of successful implementation of any vaccination program.[1] While vaccines undergo rigorous evaluations before they are licensed to the market and administered to the public, prelicensure controlled trials provide limited safety data due to their relatively small, uniform population and short duration.[2] Postlicensure vaccine safety surveillance, defined as ongoing monitoring of vaccine adverse events following immunization (AEFI) in the population, is the main element of ensuring vaccine safety.[3] AEFI may be causally linked to vaccination or may be coincidental events temporally linked to vaccine exposure. Established surveillance systems that enable voluntary (spontaneous) reporting of AEFI are the core of vaccine safety monitoring in almost all countries.[4]

These surveillance systems, however, are affected by underreporting, incomplete data, or reporting biases.[5] As such, it is valuable to supplement these systems with other data sources to enable earlier detection of AEFI, ideally close to real time, and in a more cost-effective way.

Social media has been used as a data source for public health research in many areas including forecasting, early detection, tracking trends, and discovering correlates of disease.[6] During the coronavirus disease 2019 (COVID-19) pandemic, social media data have been used to understand disease spread,[7] [8] public sentiment,[9] for analyzing government responses,[10] and for evaluating health information quality.[11] [12] The use of social media for adverse drug reaction detection is already established.[13] [14] In the vaccine domain, there is an extensive body of research on using social media for investigating vaccine opinions, attitudes, and hesitancy.[15] In contrast, the use of social media data for detection of AEFI is less explored.

A critical step in detecting AEFI in social media data is identifying posts containing text of a person reporting a personally experienced vaccine event, called a personal health mention (PHM). This is needed because unlike an AEFI reported by an individual vaccinee or a health professional to a surveillance system, there are a very large range of conversations about vaccine side effects in social media. Therefore, to correlate with reports to surveillance systems it is required to identify social media posts that contain statements about personally experienced health issues, either those of the writer or those of people they know.[16]

Other studies have focused on social media posts containing personal mentions of vaccine reactions. In one study, the detection of influenza (flu) shot vaccine reactions from Twitter was investigated by using flu shot-related keywords for collecting tweets, and then using a Natural Language Processing (NLP) classifier[17] to identify users who mention they have received a flu vaccine. They then tracked those users' tweets for subsequent vaccine reaction mentions.[18] Another recent study detecting COVID-19 vaccine reactions used reaction keywords to isolate potential AEFI, and then identified personal posts by mentions of self-related keywords, such as “I,” “my,” “mine,” “me.”[19]

In this study, we used a context-based approach for evaluating text of social media data to detect COVID-19 vaccine AEFI. We gathered potential vaccine reactions using vaccine and reaction keywords, then used NLP to identify PHMs. These techniques concentrated on the semantic and syntactic structure of the health-related social media post to identify PHM by their type of language. This approach enabled us to efficiently cast a wide net to identify PHM, compared with the other approaches we had evaluated above. We were neither limited to first identifying specific vaccinees to track subsequent posts for reaction mentions, nor to a fixed set of self-related keywords to detect personal mentions. The focus of our approach was to detect AEFI by evaluating the text of posts for the likelihood of genuine personally experienced vaccine reactions.

Objectives

In this research we set to address the following research questions:

Question 1: How can PHMs of COVID-19 vaccine reactions be identified in Twitter posts?

Question 2: Are COVID-19 vaccine reaction mentions in social media a potentially useful adjunct to traditional vaccine surveillance reporting systems for the early detection of COVID-19 vaccine safety signals?

Methods

This study implemented a two-step approach to identify PHM of COVID-19 vaccine reactions. The first step employed topic models to focus on social media conversations describing personally experienced vaccine reactions. This simplified the task of finding tweets suitable for labeling for the second step of training a classifier. We trained a modern “transformer-based” NLP text classifier using the labeled data—it was able to use the entire texts of tweets to identify those likely to contain descriptions of personally experienced vaccine reactions. We applied the classifier to the remaining data, and so obtained a data set that contained likely personal vaccine reactions, which were used for our analysis in this article.

These data were compared with COVID-19 AEFI reports to Victoria's spontaneous vaccine safety surveillance system, Surveillance of Adverse Events Following Vaccination In the Community (SAEFVIC) to determine how well aligned the proportions of reactions reporting were between the two data sources, and whether any increased difference in the tweet data reactions proportions could be potentially leveraged to enhance vaccine safety monitoring.

Data Collection

We collected 2,901,096 English tweets, posted by 1,398,970 users, from the period February 1, 2021, to January 31, 2022, using a COVID-19 vaccines and vaccine reaction keyword list. We used the Twitter academic research access Application Programming Interface (API) v2,[20] which enables an unlimited historical search into Twitter data. We used the Searchtweets-v2 wrapper to access the Twitter API v2, which handled request rate limit.[21] The search criteria required the existence of both a vaccine and a reaction word in a text. The list of keywords can be found in [Table 1]. We compiled the list of reactions based on the vaccine adverse events list historically found in SAEFVIC reports, and in liaison with other researchers in this field.[19] These included medical terms such as paresthesia and equivalent lay terms such as tingling (we used “tingl” to allow matching to any similar term). SAEFVIC is the jurisdictional vaccine safety service for the state of Victoria (population 6.5 million) and has maintained a database of AEFI reports since 2007.[22] [23]

Table 1 Twitter search keywords

COVID-19 vaccine keywords

pfizer, moderna, spikevax, astrazeneca, vaccine, vax, “the AZ,” vaxzevria, novavax, nuvaxovid, covid19vaccine, covidvaccine, covidvax, mRNA, “my shot,” “my jab,” “the jab,” “first shot,” “second shot,” booster, “1st shot,” “2nd shot,” “first dose,” “second dose”

Reaction keywords

headache, fatigue, lethargy, tired, weakness, pain, sore, hurt, aches, aching, chills, fever, nausea, drowsy, confused, dizziness, “light head,” unsteady, dizzy, reaction, syncope, faint, muscle, myalgia, joint, arthralgia, chest, myocarditis, myorcarditis, pericarditis, rash, hives, diarrhea, diarrhea, paresthesia, tingl, burning, vomiting, swollen, inflamed, cough, flush, itchy, scratchy, redness, clot, bleed, shaking, “shortness of breath,” “bad taste,” smell, abdominal, stomach, seizure, fit, numbness, drooping, allergic, anaphylaxis, period

The use of reaction keywords ([Table 1]) enabled restricting the records collected to those that were most likely to contain some mention of a reaction. The search term string was constrained by the Twitter API to a maximum of 1,024 characters, which meant we had to use a small number of search words. Although performing a search that required both a vaccine and a reaction keyword limited the data, we chose the keywords to cover a broad range of possibilities, focusing mostly on lay reaction terms. The data we collected contained many other reaction words, as reaction keywords were normally found in conjunction with other reaction words. Consequently, the data were adequate for the purpose of quantifying the volumes of these words.

Topic Modeling (Step 1)

We sampled our year's data at 2,000 records per month. These were prepared for topic modeling, which included removing stop words, expanding contractions, lemmatizing, and creating and pruning the resulting vocabulary. This process removed words that were either not frequent enough or too frequent to contribute meaningfully to topic differentiation. After eliminating posts that contained less than four words, 22,277 tweets remained. We then labeled 672 of these tweets with labels corresponding with personally experienced vaccine reactions, personally reported vaccination with no reaction mention, discussions, and news or research headlines.

These categories were to draw distinctions between PHM with and without reactions, and other common themes found in the texts. PHMs very often have a different linguistic structure to other discussions, and certainly to other types of texts such as news or research headlines, which topic models can distinguish. The annotation was independently performed by two researchers. Cohen's kappa was calculated with 0.90 intra-annotator reliability, indicating almost perfect agreement.[24]

Labeling samples this way prepared the data for evaluation by the scoring method previously described.[25] This scores models by measuring the proportion of personal vaccine reaction mentions versus the other labeled records in one or two topics, thus allowing observation of how these labels are distributed into topics during topic modeling. With this scoring approach the best model is the one with the highest proportion of personal vaccine reaction mentions in one topic.

We assessed topic models based on Latent Dirichlet Allocation[26] and Dirichlet Multinomial Mixture (DMM).[27] Our scoring method showed that the DMM model performed best at isolating PHM in tweets. We took the data from the best topic, as it was most likely to contain PHM tweets and would therefore be easier to manage for classification annotation. In other words, because the data set is most likely to contain PHM it overcomes much of the class imbalance found in the raw data, and so the data are much easier to label for genuine vaccine reactions. A previous work[28] describes the benefit of this approach for assisting with the class imbalance problem and contains a detailed analysis of classifiers that deal well with PHM in Twitter data.

Classification (Step 2)

Topic modeling isolated likely posts of vaccine-related PHM. We labeled 8,700 as either personally experienced vaccine reactions or not—4,350 records of each. Two researchers performed the labeling with Cohen's kappa intra-annotator reliability of 0.95. These were split into 7,200 training, 1,040 validation, and 260 test records, each data set also containing an equal number of each label. We fine-tuned a BERTweet-Large[29] transformer-based classifier using the training data and evaluated it with the validation and test data. The model was assessed using F1 scores against the reaction label. It achieved 0.96 on the validation data and 0.92 on the (previously unseen) test data. We then applied the best classifier model to the remaining records of the original data set, taking those identified as likely personal reaction records to add to the previously 4,350 labeled reaction records. [Table 2] shows details of the scores obtained in both data sets. We corrected labeling of some of the bleeding cohort in the classified records, which is described in [Supplementary Appendix A] (available in the online version).

Table 2 Classification scores

Data set	Precision	Recall	F1	True +ve	False +ve	False −ve	True −ve	Total records
Validation	0.970	0.946	0.958	492	15	28	505	1,040
Test	0.964	0.871	0.915	108	4	16	132	260

The final record set consisted of 2,973,032 records, with 1,021,434 assigned a positive PHM attribution and 1,951,598 assigned a negative one. The positively identified records were suitable for our subsequent analysis of probable personal reaction mentions in Twitter. Appendix B describes the process we followed to prepare the data for analysis.

Results

As we needed to compare Twitter reactions with SAEFVIC data, we grouped the Twitter reactions to be roughly equivalent to reactions found in SAEFVIC. While doing so, we also grouped similar reactions, for instance nausea with vomiting, and most of the various types of bleeding were grouped as bleeding. If we considered a particular feature might be best distinguished, then we created a separate group for them—for instance, arm pain and chest pain were separate from pain. To compare groupings between Twitter and SAEFVIC more equitably we combined some of the SAEFVIC reactions. This resulted in 26 AEFI groups in the Twitter data—see [Supplementary Appendix B] (available in the online version). We compared the proportions of these AEFI types with the proportions of reports in the top 26 groups from SAEFVIC across all COVID-19 vaccines used in Australia during the data period. These were Vaxzevria (manufacturer AstraZeneca), Spikevax (Moderna), and Comirnaty (Pfizer). With SAEFVIC data we counted distinct reports per group, whereas for Twitter we counted distinct Twitter Author IDs, as Twitter authors tend to post multiple times, often for the same event—thus counting the author compensated for this repetitiveness. For presenting our comparisons in charts we show only the top 20 of both data sets.

The first 10 groups per COVID-19 vaccine brand from each data source were comparable, though the group order sometimes changed. The top 6 groups from both sides mostly related to pain, headache, fatigue, nausea, and vomiting. Twitter users had a lot to say about arm pain, which reflects the commonplace nature of this reaction, but which tends to not feature so significantly in reports to SAEFVIC. This is probably because vaccinees are unlikely to report minor and common or expected reactions such as arm pain to SAEFVIC.

For Vaxzevria, after manual verification and label corrections ([Supplementary Appendix A], available in the online version), bleeding-related tweets were proportionally three times higher than bleeding-related reports in the SAEFVIC data. For Spikevax, there was significant SAEFVIC reporting of chest pain, and a lesser extent for Comirnaty; chest pain did not have comparable proportions in the tweets. Charts comparing the top 20 vaccine reactions reporting from both systems are shown in the following figures.

[Fig. 1] is the Vaxzevria comparison. Notably, bleeding is very prominent in Twitter conversations, the sixth most prolific topic and threefold in proportion to SAEFVIC reports received.

Fig. 1 Vaxzevria counts proportions.

In [Fig. 2], the Spikevax comparison, chest pain and myocarditis and pericarditis are significant reactions reported in the SAEFVIC data, whereas Twitter data has little mention of them. Tweets largely report common and expected reactions. However, bleeding and menstrual issues appear higher in the Twitter data than they do in the SAEFVIC data.

Fig. 2 Spikevax counts proportions.

[Fig. 3] compares Comirnaty. Chest pain, myocarditis, and pericarditis are not so prominently reported for Comirnaty in Twitter compared with Spikevax. Reports of vaccine reactions in the Twitter data are reasonably prominent—these include allergic reactions and anaphylaxis.

Fig. 3 Comirnaty counts proportions.

[Fig. 4] shows the same data from the three Twitter charts above but concatenated together. The percentages are the same as in the previous figures—that is, they represent the percentage of the topic for the brand of COVID-19 vaccine, not a percentage within the row on the chart. The top 26 Twitter reaction groups are shown, which has increased the rows in the chart from 20 to 26. The proportion of bleeding-related conversations in the Vaxzevria data are four times of that found in the other vaccines.

Fig. 4 Twitter vaccines and reactions.

The same vaccines comparison for SAEFVIC data are presented in [Fig. 5]. There are 28 rows in the chart to account for all the previously reported top 20 AEFI from each of the vaccines in the SAEFVIC data. Consequently, in some cases the values are zero, where reactions cannot be found for a vaccine in the combined data.

Fig. 5 Surveillance of Adverse Events Following Vaccination In the Community (SAEFVIC) vaccines and reactions.

Discussion

The use of social media for monitoring large populations and collecting epidemic information has been emphasized by the World Health Organization[30] and is an important additional data source for vaccine safety monitoring where vaccines have been introduced without the usual long-term safety and efficacy studies, such as with the COVID-19 vaccines.

Our first research question was to determine if PHMs of COVID-19 vaccine reactions can be identified from social media posts. To answer this, our approach was to first use vaccine and reaction keywords to obtain a set of Twitter posts. We then used a two-stage NLP approach to identify vaccine-related PHM in the posts, positively answering the research question. Research question 2 was to establish that vaccine reaction mentions in social media could be a complementary resource for early detection of COVID-19 vaccine safety issues. This was addressed by comparing the PHM posts with data obtained from a traditional vaccine surveillance system to determine their potential usefulness as an additional monitoring data source. This showed a strong correlation with the existing SAEFVIC spontaneous vaccine surveillance reporting system. In addition, our approach indicated that social media posts could be showing vaccinee concerns that are not as prominent in the established reporting system.

Principal Findings

We identified a correlation between the proportions of COVID-19 vaccine reactions in Twitter posts containing vaccine-related PHMs and SAEFVIC AEFI surveillance reports. Common and expected reactions in the top six topics generally match between the data sets, although usually in a different order.

There is significant reporting about bleeding issues in the Twitter data, especially in association with Vaxzevria, but also in the other vaccine brands. Although some of this is due to the model inaccurately identifying conversations that are not actually PHM of vaccine reactions, a manual examination of the data showed that genuine bleeding reports are a considerably large proportion, compared with SAEFVIC reporting. On the other hand, SAEFVIC had a greater proportion of heart-related reports compared with Twitter. This suggests that social media can become a timely and valuable data source for identifying emerging issues that are not captured through other data collection methods, and that our approach of using social media data to detect vaccine reactions could become an important adjunct to the data sources SAEFVIC and other vaccine surveillance systems use. For example, we detected tweets that contained mentions of bleeding and clotting issues dated prior to the European Medicine Agency announcement on March 10, 2021 that they were reviewing thromboembolic (bleeding) events associated with the AstraZeneca vaccine.[31] There are tweets about clotting and bleeding issues associated with Vaxzevria from the very beginning of our reporting period, which was February 1, 2021. These are shown in [Supplementary Appendix C] (available in the online version). Additionally, given that underreporting of common reactions to SAEFVIC and other vaccine surveillance systems is a proven phenomenon,[32] social media surveillance could be used to detect underreported common reactions for analysis of changes in frequency, which could prompt further investigation, for example, by SAEFVIC.

Upon further examination of the tweets, we also found strong statements and opinions about the COVID-19 vaccines. These ranged from very positive expressions of gratitude for receiving a vaccination despite some side effects, to alarm and anxiety about the reactions being experienced by the tweet authors or their family members.[33] Although a separate thematic and temporal analysis needs to be conducted, an observation of many conversations in the temporally latter part of the data show an emerging perception of the lack of benefit of boosters, with recipients reporting either yet another round of painful side effects or the frustration of experiencing a COVID infection despite their vaccination status. These indicate an association between side effects and people's perceptions about vaccines,[33] for further investigation. It should also be noted, that as per the nature of PHM data, some Twitter authors are not vaccinated, but are reporting about the experience of relatives who have been. In this way the tweets resemble how SAEFVIC reports can be submitted by caregivers or health professionals on behalf of a vaccinee who has had a reaction. There is a sense that Twitter is very much functioning as an online “town square” to vent opinions. Examples of tweets having an Australian context can be found in [Supplementary Appendix D] (available in the online version).

Limitations

There are limitations in using social media data for monitoring vaccine reactions. The choice of keywords for data collection impacts the amount of collected data; however, our analysis showed that reactions that were not in the list of keywords also appeared in the texts. This is because reaction mentions often accompany each other. The limitations of using keywords also applies to our exploratory analysis of reactions, that were used to identify the themes of posts—part of the reason why some reaction groups get greater prominence may be due to our identification of them with a greater number of possible keywords.

Another limitation of this study is the use of Twitter as a data source. Twitter platform affordance, which is driven by its business objectives and influenced by politics as well as platform design, will impact the user's behavior and ultimately the data. One example is the Twitter statement on combating misinformation about COVID-19[34] which could have resulted in eliminating valid concerns about COVID-19 vaccines. There is also an inherent bias in the platform's user population, which leaves out most of the general population who do not use Twitter. Mentions of AEFI such as bleeding may also be influenced by other factors, including media publicity about them. Another limitation of this study is the focus on only English language tweets.

Future Research

We aim to extend the study to (1) at data collection, either use expanded reactions lists or obtain all mentions of the vaccines of interest (regardless of any accompanying reaction mention), then retrain our PHM detection models on that data. (2) Employ entity detection techniques to identify vaccines, reactions, and subjects in the filtered posts. (3) Develop statistical measures to evaluate the correlation between reactions found in social media and vaccine surveillance reporting systems. (4) Use the findings as a basis for designing a social media surveillance application that can become a valuable adjunct to traditional reporting systems.

Conclusion

Our study has provided an initial validation step suggesting that detection of PHMs on Twitter can be used to enhance vaccine safety surveillance to improve timely responses to emerging vaccine events. Social media also provides insights into how people are experiencing the pandemic and vaccine rollout.

Clinical Relevance Statement

PHM referring to AEFI can be detected and categorized in near real-time from social media. AEFI themes correlate well with those detected from traditional spontaneous AEFI surveillance and may provide additional sensitivity for some reaction types. Social media AEFI surveillance has promise as an adjunctive mode of vaccine safety surveillance.

Multiple-Choice Questions

1. What is the most frequently discussed reaction on social media?

   • Bleeding

   • Headache

   • Arm pain

   • Reaction

   Correct Answer: The correct answer is option c.

   Explanation: arm pain is the most frequently discussed reaction on social media. Approximately 29.29% (n = 299,181) people mentioned this reaction in their tweets.

2. For which vaccine cough is reported as a reaction?

   • AstraZeneca

   • Moderna

   • Pfizer

   • None

   Correct Answer: The correct answer is option c.

   Explanation: In SAEFVIC count proportions for the Pfizer vaccine, cough is listed as a reaction with the least frequency.

3. Which model is fine-tuned for the classification process?

   • BERTweet-Large

   • LDA

   • Dirichlet Multinomial Mixture

   • PHM

   Correct Answer: the correct answer is option a.

   Explanation: BERTweet-Large is a pretrained language model for the English tweets which is very effective for NLP tasks including classification. The corpus used to pretrain BERTweet consists of 850M English tweets.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

Ethics approval for this study was granted by The Royal Children's Hospital Melbourne Human Research Ethics Committee (HREC) (project ID 85026).

• References

• 1 Black S. Recognizing the importance of vaccine confidence. EBioMedicine 2016; 12: 28-29
• 2 Jacobson RM, Adegbenro A, Pankratz VS, Poland GA. Adverse events and vaccination-the lack of power and predictability of infrequent events in pre-licensure study. Vaccine 2001; 19 (17-19): 2428-2433
• 3 Hervé C, Laupèze B, Del Giudice G, Didierlaurent AM, Tavares Da Silva F. The how's and what's of vaccine reactogenicity. NPJ Vaccines 2019; 4 (01) 39
• 4 Oliveira PMN, Lignani LK, Conceição DAD. et al. Surveillance of adverse events following immunization in the late 2010s: an overview of the importance, tools, and challenges [in Portuguese]. Cad Saude Publica 2020; 36 (Suppl. 02) e00182019
• 5 Pal SN, Duncombe C, Falzon D, Olsson S. WHO strategy for collecting safety data in public health programmes: complementing spontaneous reporting systems. Drug Saf 2013; 36 (02) 75-81
• 6 Paul MJ, Dredze M. Social monitoring for public health. Synth Lect Inf Concepts Retr Serv 2017; 9 (05) 1-183
• 7 Wilson AE, Lehmann CU, Saleh SN, Hanna J, Medford RJ. Social media: a new tool for outbreak surveillance. Antimicrob Steward Healthc Epidemiol 2021; 1 (01) e50
• 8 Qin L, Sun Q, Wang Y. et al. Prediction of number of cases of 2019 novel coronavirus (COVID-19) using social media search index. Int J Environ Res Public Health 2020; 17 (07) E2365
• 9 Medford RJ, Saleh SN, Sumarsono A, Perl TM, Lehmann CU. An “Infodemic”: leveraging high-volume twitter data to understand early public sentiment for the coronavirus disease 2019 outbreak. Open Forum Infect Dis 2020; 7 (07) ofaa258
• 10 Tsao SF, Chen H, Tisseverasinghe T, Yang Y, Li L, Butt ZA. What social media told us in the time of COVID-19: a scoping review. Lancet Digit Health 2021; 3 (03) e175-e194
• 11 Lanier HD, Diaz MI, Saleh SN, Lehmann CU, Medford RJ. Analyzing COVID-19 disinformation on Twitter using the hashtags #scamdemic and #plandemic: retrospective study. PLoS One 2022; 17 (06) e0268409
• 12 Cinelli M, Quattrociocchi W, Galeazzi A. et al. The COVID-19 social media infodemic. Sci Rep 2020; 10 (01) 16598
• 13 Sarker A, Ginn R, Nikfarjam A. et al. Utilizing social media data for pharmacovigilance: a review. J Biomed Inform 2015; 54: 202-212
• 14 Lardon J, Abdellaoui R, Bellet F. et al. Adverse drug reaction identification and extraction in social media: a scoping review. J Med Internet Res 2015; 17 (07) e171
• 15 Karafillakis E, Martin S, Simas C. et al. Methods for social media monitoring related to vaccination: systematic scoping review. JMIR Public Health Surveill 2021; 7 (02) e17149
• 16 Yin Z, Fabbri D, Rosenbloom ST, Malin B. A scalable framework to detect personal health mentions on Twitter. J Med Internet Res 2015; 17 (06) e138
• 17 Yu H, Ho C, Juan Y, Lin C. Libshorttext: a library for short-text classification and analysis. Rapport interne, Department of Computer Science, National Taiwan University Software. Published online 2013. Accessed November 25, 2022, at: http://www csie ntu edu tw/cjlin/libshorttext
• 18 Wang J, Zhao L. Multi-instance domain adaptation for vaccine adverse event detection. 2018: 97-106
• 19 Lian AT, Du J, Tang L. Using a machine learning approach to monitor COVID-19 vaccine adverse events (VAE) from Twitter data. Vaccines (Basel) 2022; 10 (01) 103
• 20 Twitter. Twitter API Academic Research. Published 2022. Accessed September 2, 2022, at: https://developer.twitter.com/en/products/twitter-api/academic-research
• 21 Pigott F, Kolb J, Montague J, Gonzales A, Moffitt J. Searchtweets, Wrapper for Twitter API v2 search endpoints. Published 2021. Accessed September 2, 2022, at: https://pypi.org/project/searchtweets-v2/
• 22 Clothier HJ, Crawford NW, Kempe A, Buttery JP. Surveillance of adverse events following immunisation: the model of SAEFVIC, Victoria. Commun Dis Intell Q Rep 2011; 35 (04) 294-298
• 23 SAEFVIC. Surveillance of adverse events following vaccination in the community. Published 2022. Accessed November 25, 2022, at: https://www.mcri.edu.au/research/research-areas/infection-and-immunity/saefvic
• 24 McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012; 22 (03) 276-282
• 25 Khademi Habibabadi S, Haghighi PD, Habibabadi SK, Haghighi PD, Khademi S, Haghighi PD. Topic modelling for identification of vaccine reactions in Twitter. ACM International Conference Proceeding Series. Published online 2019:31. Accessed November 25, 2022 at: https://dl.acm.org/doi/abs/10.1145/3290688.3290735
• 26 Řehůřek R, Sojka P. Software Framework for Topic Modelling with Large Corpora. Proceedings of LREC 2010 Workshop New Challenges. Published online 2010. Accessed November 25, 2022, at: http://www.muni.cz/research/publications/884893
• 27 Nguyen DQ. jLDADMM: A Java package for the LDA and DMM topic models. 2018 (Dmm):1–5. Accessed November 25, 2022, at: http://arxiv.org/abs/1808.03835
• 28 Mining of Twitter Conversations: 2-Phase Classification Study. JMIR Med Inform 2022; 10 (06) E34305
• 29 Nguyen DQ, Vu T, Nguyen AT. BERTweet: a pre-trained language model for English Tweets. arXiv preprint arXiv:200510200. Published online 2020
• 30 WHO. . COVID-19 Vaccines: Safety Surveillance Manual; Published online 2021. Accessed December 3, 2022 at: https://www.who.int/publications/i/item/9789240032781
• 31 Agency EM. COVID-19 Vaccine AstraZeneca: PRAC preliminary view suggests no specific issue with batch used in Austria | European Medicines Agency. Published 2021. Accessed April 29, 2022, at: https://www.ema.europa.eu/en/news/covid-19-vaccine-astrazeneca-prac-preliminary-view-suggests-no-specific-issue-batch-used-austria
• 32 Crawford NW, Clothier H, Hodgson K, Selvaraj G, Easton ML, Buttery JP. Active surveillance for adverse events following immunization. Expert Rev Vaccines 2014; 13 (02) 265-276
• 33 Danchin M, Buttery J. COVID-19 vaccine hesitancy: a unique set of challenges. Intern Med J 2021; 51 (12) 1987-1989
• 34 Niemiec E. COVID-19 and misinformation: is censorship of social media a remedy to the spread of medical misinformation?. EMBO Rep 2020; 21 (11) e51420

Publication History

Received: 11 July 2022

Accepted: 25 October 2022

Accepted Manuscript online:
09 November 2022

Article published online:
04 January 2023