  Model selection on validation data is an essential step in machine learning.
While the mixing of data between training and validation is considered taboo,
practitioners often violate it to increase performance. Here, we offer a
simple, practical method for using the validation set for training, which
allows for a continuous, controlled trade-off between performance and
overfitting of model selection. We define the notion of
on-average-validation-stable algorithms as one in which using small portions of
validation data for training does not overfit the model selection process. We
then prove that stable algorithms are also validation stable. Finally, we
demonstrate our method on the MNIST and CIFAR-10 datasets using stable
algorithms as well as state-of-the-art neural networks. Our results show
significant increase in test performance with a minor trade-off in bias
admitted to the model selection process.
