  This paper investigates the central role played by the Hamiltonian in
continuous-time nonlinear optimal control problems. We show that the strict
convexity of the Hamiltonian in the control variable is a sufficient condition
for the existence of a unique optimal trajectory, and the
nonlinearity/non-convexity of the dynamics and the cost are immaterial. The
analysis is extended to discrete-time problems, revealing that discretization
destroys the convex Hamiltonian structure, leading to multiple spurious optima,
unless the time discretization is sufficiently small. We present simulated
results comparing the "indirect" Iterative Linear Quadratic Regulator (iLQR)
and the "direct" Sequential Quadratic Programming (SQP) approach for solving
the optimal control problem for the cartpole and pendulum models to validate
the theoretical analysis. Results show that the ILQR always converges to the
"globally" optimum solution while the SQP approach gets stuck in spurious
minima given multiple random initial guesses for a time discretization that is
insufficiently small, while both converge to the same unique solution if the
discretization is sufficiently small.
