Unlike other articles, I found the wikipedia entry for this subject unreadable for a non-math person (like me).

I understood the basic idea, that you favor models with fewer rules. What I don't get is how do you get from a set of rules to a 'regularization score' which you can use to sort the models from least to most overfit.

Can you describe a simple regularization method?

I'm interested in the context of analyzing statistical trading systems. It would be great if you could describe if/how I can apply regularization to analyze the following two predictive models:

Model 1 - price going up when:


exp_moving_avg(price, period=50) > exp_moving_avg(price, period=200)


Model 2 - price going up when:


price[n] < price[n-1] 10 times in a row
exp_moving_avg(price, period=200) going up


But I'm more interested in getting a feeling for how you do regularization. So if you know better models for explaining it please do.

In simple terms, regularization is tuning or selecting the preferred level of model complexity so your models are better at predicting (generalizing). If you don't do this your models may be too complex and overfit or too simple and underfit, either way giving poor predictions.

If you least-squares fit a complex model to a small set of training data you will probably overfit, this is the most common situation. The optimal complexity of the model depends on the sort of process you are modeling and the quality of the data, so there is no a-priori correct complexity of a model.

To regularize you need 2 things:
A way of testing how good your models are at prediction, for example using cross-validation or a set of validation data (you can't use the fitting error for this).
A tuning parameter which lets you change the complexity or smoothness of the model, or a selection of models of differing complexity/smoothness.

Basically you adjust the complexity parameter (or change the model) and find the value which gives the best model predictions.

Note that the optimized regularization error will not be an accurate estimate of the overall prediction error so after regularization you will finally have to use an additional validation dataset or perform some additional statistical analysis to get an unbiased prediction error.

An alternative to using (cross-)validation testing is to use Bayesian Priors or other methods to penalize complexity or non-smoothness, but these require more statistical sophistication and knowledge of the problem and model features.
