  We use a previously developed scattering-amplitudes-based framework for
determining two-body Hamiltonians for generic binary systems with arbitrary
spin $S$. By construction this formalism bypasses difficulties with unphysical
singularities or higher-time derivatives. This framework has been previously
used to obtain the exact velocity dependence of the $\mathcal O(G^2)$
quadratic-in-spin two-body Hamiltonian. We first evaluate the $S^3$ scattering
angle and two-body Hamiltonian at this order in $G$, including not only all
operators corresponding to the usual worldline operators, but also an
additional set due to an interesting subtlety. We then evaluate $S^4$ and $S^5$
contributions at $\mathcal O(G^2)$ which we confirm by comparing against
aligned-spin results. We conjecture that a certain shift symmetry together with
a constraint on the high-energy growth of the scattering amplitude specify the
Wilson coefficients for the Kerr black hole to all orders in the spin and
confirm that they reproduce the previously-obtained results through $S^4$.
