  We revisit the inverted pendulum problem with the goal of understanding and
computing the true optimal value function. We start with an observation that
the true optimal value function must be nonsmooth ($i.e.$, not globally $C^1$)
due to the symmetry of the problem. We then give a result that can certify the
optimality of a candidate $\textit{piece-wise}$ $C^1$ value function. Further,
for a candidate value function obtained via numerical approximation, we provide
a bound of suboptimality based on its Hamilton-Jacobi-Bellman (HJB) equation
residuals. Inspired by Holzhuter (2004), we then design an algorithm that
solves backward the Pontryagin's minimum principle (PMP) ODE from terminal
conditions provided by the locally optimal LQR value function. This numerical
procedure leads to a piece-wise $C^1$ value function whose nonsmooth region
contains periodic $\textit{spiral lines}$ and smooth regions attain HJB
residuals about $10^{-4}$, hence certified to be the optimal value function up
to minor numerical inaccuracies. This optimal value function checks the power
of optimality: (i) it sits above a polynomial lower bound; (ii) its induced
controller globally swings up and stabilizes the pendulum, and (iii) attains
lower trajectory cost than baseline methods such as energy shaping, model
predictive control (MPC), and proximal policy optimization (with MPC attaining
almost the same cost). We conclude by distilling the optimal value function
into a simple neural network. Our code is avilable in
https://github.com/ComputationalRobotics/InvertedPendulumOptimalValue.
