  We present ShinRL, an open-source library specialized for the evaluation of
reinforcement learning (RL) algorithms from both theoretical and practical
perspectives. Existing RL libraries typically allow users to evaluate practical
performances of deep RL algorithms through returns. Nevertheless, these
libraries are not necessarily useful for analyzing if the algorithms perform as
theoretically expected, such as if Q learning really achieves the optimal Q
function. In contrast, ShinRL provides an RL environment interface that can
compute metrics for delving into the behaviors of RL algorithms, such as the
gap between learned and the optimal Q values and state visitation frequencies.
In addition, we introduce a flexible solver interface for evaluating both
theoretically justified algorithms (e.g., dynamic programming and tabular RL)
and practically effective ones (i.e., deep RL, typically with some additional
extensions and regularizations) in a consistent fashion. As a case study, we
show that how combining these two features of ShinRL makes it easier to analyze
the behavior of deep Q learning. Furthermore, we demonstrate that ShinRL can be
used to empirically validate recent theoretical findings such as the effect of
KL regularization for value iteration and for deep Q learning, and the
robustness of entropy-regularized policies to adversarial rewards. The source
code for ShinRL is available on GitHub: https://github.com/omron-sinicx/ShinRL.
