  Estimators derived from score functions that are not the likelihood are in
wide use in practical and modern applications. Their regularization is often
carried by pseudo-posterior estimation, equivalently by adding penalty to the
score function. We argue that this approach is suboptimal, and propose a
two-staged alternative involving estimation of a new score function which
better approximates the true likelihood for the purpose of regularization. Our
approach typically identifies with maximum a-posteriori estimation if the
original score function is in fact the likelihood. We apply our theory to
fitting ordinary least squares (OLS) under contemporaneous exogeneity, a
setting appearing often in time series and in which OLS is the estimator of
choice by practitioners.
