Logic often states that by overfitting a model, its capacity to generalize is limited, though this might only mean that overfitting stops a model from improving after a certain complexity. Does overfitting cause models to become worse regardless of the complexity of data, and if so, why is this the case?



Related: Followup to the question above, "When is a Model Underfitted?"

Overfitting is empirically bad.  Suppose you have a data set which you split in two, test and training.  An overfitted model is one that performs much worse on the test dataset than on training dataset.  It is often observed that models like that also in general perform worse on additional (new) test datasets than models which are not overfitted.  

One way to understand that intuitively is that a model may use some relevant parts of the data (signal) and some irrelevant parts (noise).  An overfitted model uses more of the noise, which increases its performance in the case of known noise (training data) and decreases its performance in the case of novel noise (test data).  The difference in performance between training and test data indicates how much noise the model picks up; and picking up noise directly translates into worse performance on test data (including future data).

Summary: overfitting is bad by definition, this has not much to do with either complexity or ability to generalize, but rather has to do with mistaking noise for signal.

P.S. On the "ability to generalize" part of the question, it is very possible to have a model which has inherently limited ability to generalize due to the structure of the model (for example linear SVM, ...) but is still prone to overfitting.  In a sense overfitting is just one way that generalization may fail.
