  Understanding the encoded representation of Deep Neural Networks (DNNs) has
been a fundamental yet challenging objective. In this work, we focus on two
possible directions for analyzing representations of DNNs by studying simple
image classification tasks. Specifically, we consider \textit{On-Off pattern}
and \textit{PathCount} for investigating how information is stored in deep
representations. On-off pattern of a neuron is decided as `on' or `off'
depending on whether the neuron's activation after ReLU is non-zero or zero.
PathCount is the number of paths that transmit non-zero energy from the input
to a neuron. We investigate how neurons in the network encodes information by
replacing each layer's activation with On-Off pattern or PathCount and
evaluating its effect on classification performance. We also examine
correlation between representation and PathCount. Finally, we show a possible
way to improve an existing DNN interpretation method, Class Activation Map
(CAM), by directly utilizing On-Off or PathCount.
