  Instruction-finetuned Large Language Models exhibit unprecedented Natural
Language Understanding capabilities. Recent work has been exploring political
biases and political reasoning capabilities in LLMs, mainly scoped in the US
context. In light of the recent 2024 European Parliament elections, we are
investigating if LLMs can be used as Voting Advice Applications (VAAs). We
audit MISTRAL and MIXTRAL models and evaluate their accuracy in predicting the
stance of political parties based on the latest "EU and I" voting assistance
questionnaire. Furthermore, we explore alternatives to improve models'
performance by augmenting the input context via Retrieval-Augmented Generation
(RAG) relying on web search, and Self-Reflection using staged conversations
that aim to re-collect relevant content from the model's internal memory. We
find that MIXTRAL is highly accurate with an 82% accuracy on average.
Augmenting the input context with expert-curated information can lead to a
significant boost of approx. 9%, which remains an open challenge for automated
approaches.
