  Recently deep neural networks demonstrate competitive performances in
classification and regression tasks for many temporal or sequential data.
However, it is still hard to understand the classification mechanisms of
temporal deep neural networks. In this paper, we propose two new frameworks to
visualize temporal representations learned from deep neural networks. Given
input data and output, our algorithm interprets the decision of temporal neural
network by extracting highly activated periods and visualizes a sub-sequence of
input data which contributes to activate the units. Furthermore, we
characterize such sub-sequences with clustering and calculate the uncertainty
of the suggested type and actual data. We also suggest Layer-wise Relevance
from the output of a unit, not from the final output, with backward Monte-Carlo
dropout to show the relevance scores of each input point to activate units with
providing a visual representation of the uncertainty about this impact.
