  Following recent policy changes by X (Twitter) and other social media
platforms, user interaction data has become increasingly difficult to access.
These restrictions are impeding robust research pertaining to social and
political phenomena online, which is critical due to the profound impact social
media platforms may have on our societies. Here, we investigate the reliability
of polarization measures obtained from different samples of social media data
by studying the structural polarization of the Polish political debate on
Twitter over a 24-hour period. First, we show that the political discussion on
Twitter is only a small subset of the wider Twitter discussion. Second, we find
that large samples can be representative of the whole political discussion on a
platform, but small samples consistently fail to accurately reflect the true
structure of polarization online. Finally, we demonstrate that keyword-based
samples can be representative if keywords are selected with great care, but
that poorly selected keywords can result in substantial political bias in the
sampled data. Our findings demonstrate that it is not possible to measure
polarization in a reliable way with small, sampled datasets, highlighting why
the current lack of research data is so problematic, and providing insight into
the practical implementation of the European Union's Digital Service Act which
aims to improve researchers' access to social media data.
