  Standard methods in supervised learning separate training and prediction: the
model is fit independently of any test points it may encounter. However, can
knowledge of the next test point $\mathbf{x}_{\star}$ be exploited to improve
prediction accuracy? We address this question in the context of linear
prediction, showing how techniques from semi-parametric inference can be used
transductively to combat regularization bias. We first lower bound the
$\mathbf{x}_{\star}$ prediction error of ridge regression and the Lasso,
showing that they must incur significant bias in certain test directions. We
then provide non-asymptotic upper bounds on the $\mathbf{x}_{\star}$ prediction
error of two transductive prediction rules. We conclude by showing the efficacy
of our methods on both synthetic and real data, highlighting the improvements
single point transductive prediction can provide in settings with distribution
shift.
