  This research investigates whether OpenAI's GPT-4, a state-of-the-art large
language model, can accurately classify the political bias of news sources
based solely on their URLs. Given the subjective nature of political labels,
third-party bias ratings like those from Ad Fontes Media, AllSides, and Media
Bias/Fact Check (MBFC) are often used in research to analyze news source
diversity. This study aims to determine if GPT-4 can replicate these human
ratings on a seven-degree scale ("far-left" to "far-right"). The analysis
compares GPT-4's classifications against MBFC's, and controls for website
popularity using Open PageRank scores. Findings reveal a high correlation
($\text{Spearman's } \rho = .89$, $n = 5,877$, $p < 0.001$) between GPT-4's and
MBFC's ratings, indicating the model's potential reliability. However, GPT-4
abstained from classifying approximately $\frac{2}{3}$ of the dataset,
particularly less popular and less biased sources. The study also identifies a
slight leftward skew in GPT-4's classifications compared to MBFC's. The
analysis suggests that while GPT-4 can be a scalable, cost-effective tool for
political bias classification of news websites, but its use should complement
human judgment to mitigate biases. Further research is recommended to explore
the model's performance across different settings, languages, and additional
datasets.
