  With the increasing prevalence of probabilistic programming languages,
Hamiltonian Monte Carlo (HMC) has become the mainstay of applied Bayesian
inference. However HMC still struggles to sample from densities with multiscale
geometry: a large step size is needed to efficiently explore low curvature
regions while a small step size is needed to accurately explore high curvature
regions. We introduce the delayed rejection generalized HMC (DR-G-HMC) sampler
that overcomes this challenge by employing dynamic step size selection,
inspired by differential equation solvers. In a single sampling iteration,
DR-G-HMC sequentially makes proposals with geometrically decreasing step sizes
if necessary. This simulates Hamiltonian dynamics with increasing fidelity
that, in high curvature regions, generates proposals with a higher chance of
acceptance. DR-G-HMC also makes generalized HMC competitive by decreasing the
number of rejections which otherwise cause inefficient backtracking and
prevents directed movement. We present experiments to demonstrate that DR-G-HMC
(1) correctly samples from multiscale densities, (2) makes generalized HMC
methods competitive with the state of the art No-U-Turn sampler, and (3) is
robust to tuning parameters.
