  Explainability techniques are crucial in gaining insights into the reasons
behind the predictions of deep learning models, which have not yet been applied
to chemical language models. We propose an explainable AI technique that
attributes the importance of individual atoms towards the predictions made by
these models. Our method backpropagates the relevance information towards the
chemical input string and visualizes the importance of individual atoms. We
focus on self-attention Transformers operating on molecular string
representations and leverage a pretrained encoder for finetuning. We showcase
the method by predicting and visualizing solubility in water and organic
solvents. We achieve competitive model performance while obtaining
interpretable predictions, which we use to inspect the pretrained model.
