  Neural networks (NN) are considered as black-boxes due to the lack of
explainability and transparency of their decisions. This significantly hampers
their deployment in environments where explainability is essential along with
the accuracy of the system. Recently, significant efforts have been made for
the interpretability of these deep networks with the aim to open up the
black-box. However, most of these approaches are specifically developed for
visual modalities. In addition, the interpretations provided by these systems
require expert knowledge and understanding for intelligibility. This indicates
a vital gap between the explainability provided by the systems and the novice
user. To bridge this gap, we present a novel framework i.e. Time-Series
eXplanation (TSXplain) system which produces a natural language based
explanation of the decision taken by a NN. It uses the extracted statistical
features to describe the decision of a NN, merging the deep learning world with
that of statistics. The two-level explanation provides ample description of the
decision made by the network to aid an expert as well as a novice user alike.
Our survey and reliability assessment test confirm that the generated
explanations are meaningful and correct. We believe that generating natural
language based descriptions of the network's decisions is a big step towards
opening up the black-box.
