  Developing machine learning models to characterize political polarization on
online social media presents significant challenges. These challenges mainly
stem from various factors such as the lack of annotated data, presence of noise
in social media datasets, and the sheer volume of data. The common research
practice typically examines the biased structure of online user communities for
a given topic or qualitatively measuring the impacts of polarized topics on
social media. However, there is limited work focusing on analyzing polarization
at the ground-level, specifically in the social media posts themselves. Such
existing analysis heavily relies on annotated data, which often requires
laborious human labeling, offers labels only to specific problems, and lacks
the ability to determine the near-future bias state of a social media
conversations. Understanding the degree of political orientation conveyed in
social media posts is crucial for quantifying the bias of online user
communities and investigating the spread of polarized content. In this work, we
first introduce two heuristic methods that leverage on news media bias and post
content to label social media posts. Next, we compare the efficacy and quality
of heuristically labeled dataset with a randomly sampled human-annotated
dataset. Additionally, we demonstrate that current machine learning models can
exhibit improved performance in predicting political orientation of social
media posts, employing both traditional supervised learning and few-shot
learning setups. We conduct experiments using the proposed heuristic methods
and machine learning approaches to predict the political orientation of posts
collected from two social media forums with diverse political ideologies: Gab
and Twitter.
