  The rapid advancement of artificial intelligence has led to significant
improvements in automated decision-making. However, the increased performance
of models often comes at the cost of explainability and transparency of their
decision-making processes. In this paper, we investigate the capabilities of
large language models to explain decisions, using football refereeing as a
testing ground, given its decision complexity and subjectivity. We introduce
the Explainable Video Assistant Referee System, X-VARS, a multi-modal large
language model designed for understanding football videos from the point of
view of a referee. X-VARS can perform a multitude of tasks, including video
description, question answering, action recognition, and conducting meaningful
conversations based on video content and in accordance with the Laws of the
Game for football referees. We validate X-VARS on our novel dataset,
SoccerNet-XFoul, which consists of more than 22k video-question-answer triplets
annotated by over 70 experienced football referees. Our experiments and human
study illustrate the impressive capabilities of X-VARS in interpreting complex
football clips. Furthermore, we highlight the potential of X-VARS to reach
human performance and support football referees in the future.
