  The proliferation of deep neural networks in various domains has seen an
increased need for the interpretability of these models, especially in
scenarios where fairness and trust are as important as model performance. A lot
of independent work is being carried out to: i) analyze what linguistic and
non-linguistic knowledge is learned within these models, and ii) highlight the
salient parts of the input. We present NxPlain, a web application that provides
an explanation of a model's prediction using latent concepts. NxPlain discovers
latent concepts learned in a deep NLP model, provides an interpretation of the
knowledge learned in the model, and explains its predictions based on the used
concepts. The application allows users to browse through the latent concepts in
an intuitive order, letting them efficiently scan through the most salient
concepts with a global corpus level view and a local sentence-level view. Our
tool is useful for debugging, unraveling model bias, and for highlighting
spurious correlations in a model. A hosted demo is available here:
https://nxplain.qcri.org.
