  Sharing of anti-vaccine posts on social media, including misinformation
posts, has been shown to create confusion and reduce the publics confidence in
vaccines, leading to vaccine hesitancy and resistance. Recent years have
witnessed the fast rise of such anti-vaccine posts in a variety of linguistic
and visual forms in online networks, posing a great challenge for effective
content moderation and tracking. Extending previous work on leveraging textual
information to understand vaccine information, this paper presents Insta-VAX, a
new multi-modal dataset consisting of a sample of 64,957 Instagram posts
related to human vaccines. We applied a crowdsourced annotation procedure
verified by two trained expert judges to this dataset. We then bench-marked
several state-of-the-art NLP and computer vision classifiers to detect whether
the posts show anti-vaccine attitude and whether they contain misinformation.
Extensive experiments and analyses demonstrate the multimodal models can
classify the posts more accurately than the uni-modal models, but still need
improvement especially on visual context understanding and external knowledge
cooperation. The dataset and classifiers contribute to monitoring and tracking
of vaccine discussions for social scientific and public health efforts in
combating the problem of vaccine misinformation.
