  Large language models (LLMs) are increasingly being utilised across a range
of tasks and domains, with a burgeoning interest in their application within
the field of journalism. This trend raises concerns due to our limited
understanding of LLM behaviour in this domain, especially with respect to
political bias. Existing studies predominantly focus on LLMs undertaking
political questionnaires, which offers only limited insights into their biases
and operational nuances. To address this gap, our study establishes a new
curated dataset that contains 2,100 human-written articles and utilises their
descriptions to generate 56,700 synthetic articles using nine LLMs. This
enables us to analyse shifts in properties between human-authored and
machine-generated articles, with this study focusing on political bias,
detecting it using both supervised models and LLMs. Our findings reveal
significant disparities between base and instruction-tuned LLMs, with
instruction-tuned models exhibiting consistent political bias. Furthermore, we
are able to study how LLMs behave as classifiers, observing their display of
political bias even in this role. Overall, for the first time within the
journalistic domain, this study outlines a framework and provides a structured
dataset for quantifiable experiments, serving as a foundation for further
research into LLM political bias and its implications.
