  We introduce an approach for understanding control policies represented as
recurrent neural networks. Recent work has approached this problem by
transforming such recurrent policy networks into finite-state machines (FSM)
and then analyzing the equivalent minimized FSM. While this led to interesting
insights, the minimization process can obscure a deeper understanding of a
machine's operation by merging states that are semantically distinct. To
address this issue, we introduce an analysis approach that starts with an
unminimized FSM and applies more-interpretable reductions that preserve the key
decision points of the policy. We also contribute an attention tool to attain a
deeper understanding of the role of observations in the decisions. Our case
studies on 7 Atari games and 3 control benchmarks demonstrate that the approach
can reveal insights that have not been previously noticed.
