  We study partially linear models in settings where observations are arranged
in independent groups but may exhibit within-group dependence. Existing
approaches estimate linear model parameters through weighted least squares,
with optimal weights (given by the inverse covariance of the response,
conditional on the covariates) typically estimated by maximising a (restricted)
likelihood from random effects modelling or by using generalised estimating
equations. We introduce a new 'sandwich loss' whose population minimiser
coincides with the weights of these approaches when the parametric forms for
the conditional covariance are well-specified, but can yield arbitrarily large
improvements in linear parameter estimation accuracy when they are not. Under
relatively mild conditions, our estimated coefficients are asymptotically
Gaussian and enjoy minimal variance among estimators with weights restricted to
a given class of functions, when user-chosen regression methods are used to
estimate nuisance functions. We further expand the class of functional forms
for the weights that may be fitted beyond parametric models by leveraging the
flexibility of modern machine learning methods within a new gradient boosting
scheme for minimising the sandwich loss. We demonstrate the effectiveness of
both the sandwich loss and what we call 'sandwich boosting' in a variety of
settings with simulated and real-world data.
