  We describe an approach to improving model fitting and model generalization
that considers the entropy of distributions of modelling residuals. We use
simple simulations to demonstrate the observational signatures of overfitting
on ordered sequences of modelling residuals, via the autocorrelation and power
spectral density. These results motivate the conclusion that, as commonly
applied, the least squares method assumes too much when it assumes that
residuals are uncorrelated for all possible models or values of the model
parameters. We relax these too-stringent assumptions in favour of imposing an
entropy prior on the (unknown, model-dependent, but potentially marginalizable)
distribution function for residuals. We recommend a simple extension to the
Mean Squared Error loss function that approximately incorporates this prior and
can be used immediately for modelling applications where meaningfully-ordered
sequences of observations or training data can be defined.
