  Gradient-based methods enable efficient search capabilities in high
dimensions. However, in order to apply them effectively in offline optimization
paradigms such as offline Reinforcement Learning (RL) or Imitation Learning
(IL), we require a more careful consideration of how uncertainty estimation
interplays with first-order methods that attempt to minimize them. We study
smoothed distance to data as an uncertainty metric, and claim that it has two
beneficial properties: (i) it allows gradient-based methods that attempt to
minimize uncertainty to drive iterates to data as smoothing is annealed, and
(ii) it facilitates analysis of model bias with Lipschitz constants. As
distance to data can be expensive to compute online, we consider settings where
we need amortize this computation. Instead of learning the distance however, we
propose to learn its gradients directly as an oracle for first-order
optimizers. We show these gradients can be efficiently learned with
score-matching techniques by leveraging the equivalence between distance to
data and data likelihood. Using this insight, we propose Score-Guided Planning
(SGP), a planning algorithm for offline RL that utilizes score-matching to
enable first-order planning in high-dimensional problems, where zeroth-order
methods were unable to scale, and ensembles were unable to overcome local
minima. Website: https://sites.google.com/view/score-guided-planning/home
