In a supervised classification setting, is it data-snooping to perform unsupervised preprocessing on your complete dataset (train + validation), if you never look at the class labels during preprocessing?
For example, suppose you perform PCA on your complete dataset (using all datapoints, without looking at the class labels), then discard some dimensions, and then apply a supervised classifier using the new PCA predictors. Does this constitute data snooping?
Could you please comment on this issue and on the quote from Hastie et al?
Strictly speaking, one can construct an unsupervised scheme (model selection based on data inputs only, without involving the output labels) that can ruin the chances for good generalization in subsequent use of supervised learning. One example, admittedly extreme, is to take and select a model that is free to choose the values of the output on this specific set of points, but forces an output of zero outside that set. This model can then achieve in supervised learning by matching , but will obviously perform poorly out of sample since it is forced to output on any other point.
Practically speaking, 'reasonable' processing and model selection based on the data inputs doesn't normally contaminate the data much for subsequent supervised learning, so it is not unreasonable to state a heuristic rule that unsupervised processing of the data does not constitute data snooping.
In the financial data experiment given in Example 5.3 (also in Lecture 17 of the LFD course), the normalization step that constituted data snooping involved the outputs as well, since it was done on the daily returns before structuring the data into inputs and outputs, and the outputs were indeed among the daily returns.
must be "left out" before any selection or filtering steps are applied."
Interestingly, the specific example you mentioned about PCA dimension reduction is considered in Problem 9.10 of e-Chapter 9 which is posted on this forum. If you perform the experiment, you will find that this particular form of unsupervised input snooping can significantly bias your LOO-CV estimate of Eout.
Thanks to you both for replying.
It seems you disagree a bit about how significant the bias might be..?
That strikes me as a rather large bias, especially considering it's a linear model and we're only omitting 1 point.
I expected the bias to be larger in this case, but it's actually smaller.
Generally speaking, a nonparametric UFL algorithm ought to allocate representational capacity to an area of the input space in proportion to how much "statistically significant structure" is present there. Using all the data, there will be a certain amount of such structure. But inside a validation fold, even though the underlying structure is the same, there will be less statistically significant structure simply because there is insufficient data to show it all. So the in-fold UFL will deliver a more impoverished representation of the input than the 'full-data' UFL, and may miss some useful structure. Furthermore, it will do no good to simply tell the in-fold UFL to allocate the same amount of capacity that the 'full' UFL used, because it would not know where to allocate it -- there will be many places in the input that have 'almost significant' structure, but some of those will really be just noise. The advantage of the 'full' UFL is that it knows which of those areas has the real structure, and so doesn't waste capacity modeling noise (overfitting).
Ultimately, I want to know if the bias introduced by running UFL on all the data is "tolerable". I'm still not sure! Hastie et al seem to think so, but we seem to be coming to the opposite conclusion here.
Interesting observations, and yes, when there is input snooping, it can be counter-intuitive. Here is one way to interpret your result. Input snooping with LOO-CV lets you peek at the test input. This allows you to focus your learning to improve the prediction on that test input, for example by tailoring your PCA to include the test-input.
When you do 50% holdout, you are input-snooping a set of test inputs (half the data), so while I can focus the learning on these test inputs, I can do so only on `average' and so I will not be able to excessively input-snoop any particular one. With LOO-CV, you can focus on snooping on one test input at a time, which explains why the bias with LOO-CV can be higher.