  Text data are increasingly handled in an automated fashion by machine
learning algorithms. But the models handling these data are not always
well-understood due to their complexity and are more and more often referred to
as "black-boxes." Interpretability methods aim to explain how these models
operate. Among them, LIME has become one of the most popular in recent years.
However, it comes without theoretical guarantees: even for simple models, we
are not sure that LIME behaves accurately. In this paper, we provide a first
theoretical analysis of LIME for text data. As a consequence of our theoretical
findings, we show that LIME indeed provides meaningful explanations for simple
models, namely decision trees and linear models.
