  In software engineering, deep learning models are increasingly deployed for
critical tasks such as bug detection and code review. However, overfitting
remains a challenge that affects the quality, reliability, and trustworthiness
of software systems that utilize deep learning models. Overfitting can be (1)
prevented (e.g., using dropout or early stopping) or (2) detected in a trained
model (e.g., using correlation-based approaches). Both overfitting detection
and prevention approaches that are currently used have constraints (e.g.,
requiring modification of the model structure, and high computing resources).
In this paper, we propose a simple, yet powerful approach that can both detect
and prevent overfitting based on the training history (i.e., validation
losses). Our approach first trains a time series classifier on training
histories of overfit models. This classifier is then used to detect if a
trained model is overfit. In addition, our trained classifier can be used to
prevent overfitting by identifying the optimal point to stop a model's
training. We evaluate our approach on its ability to identify and prevent
overfitting in real-world samples. We compare our approach against
correlation-based detection approaches and the most commonly used prevention
approach (i.e., early stopping). Our approach achieves an F1 score of 0.91
which is at least 5% higher than the current best-performing non-intrusive
overfitting detection approach. Furthermore, our approach can stop training to
avoid overfitting at least 32% of the times earlier than early stopping and has
the same or a better rate of returning the best model.
