  The popularity of social media makes politicians use it for political
advertisement. Therefore, social media is full of electoral agitation
(electioneering), especially during the election campaigns. The election
administration cannot track the spread and quantity of messages that count as
agitation under the election code. It addresses a crucial problem, while also
uncovering a niche that has not been effectively targeted so far. Hence, we
present the first publicly open data set for detecting electoral agitation in
the Polish language. It contains 6,112 human-annotated tweets tagged with four
legally conditioned categories. We achieved a 0.66 inter-annotator agreement
(Cohen's kappa score). An additional annotator resolved the mismatches between
the first two improving the consistency and complexity of the annotation
process. The newly created data set was used to fine-tune a Polish Language
Model called HerBERT (achieving a 68% F1 score). We also present a number of
potential use cases for such data sets and models, enriching the paper with an
analysis of the Polish 2020 Presidential Election on Twitter.
