How would you describe cross-validation to someone without a data analysis background?
Consider the following situation: 


  I want to catch the subway to go to my office. My plan is to take my car, park at the subway and then take the train to go to my office. My goal is to catch the train at 8.15 am every day so that I can reach my office on time. I need to decide the following: (a) the time at which I need to leave from my home and (b) the route I will take to drive to the station.


In the above example, I have two parameters (i.e., time of departure from home and route to take to the station) and I need to choose these parameters such that I reach the station by 8.15 am. 

In order to solve the above problem I may try out different sets of 'parameters' (i.e., different combination of times of departure and route) on Mondays, Wednesdays, and Fridays, to see which combination is the 'best' one. The idea is that once I have identified the best combination I can use it every day so that I achieve my objective.

Problem of Overfitting

The problem with the above approach is that I may overfit which essentially means that the best combination I identify may in some sense may be unique to Mon, Wed and Fridays and that combination may not work for Tue and Thu. Overfitting may happen if in my search for the best combination of times and routes I exploit some aspect of the traffic situation on Mon/Wed/Fri which does not occur on Tue and Thu.

One Solution to Overfitting: Cross-Validation

Cross-validation is one solution to overfitting. The idea is that once we have identified our best combination of parameters (in our case time and route) we test the performance of that set of parameters in a different context. Therefore, we may want to test on Tue and Thu as well to ensure that our choices work for those days as well.

Extending the analogy to statistics

In statistics, we have a similar issue. We often use a limited set of data to estimate the unknown parameters we do not know. If we overfit then our parameter estimates will work very well for the existing data but not as well for when we use them in another context. Thus, cross-validation helps in avoiding the above issue of overfitting by proving us some reassurance that the parameter estimates are not unique to the data we used to estimate them.

Of course, cross validation is not perfect. Going back to our example of the subway, it can happen that even after cross-validation, our best choice of parameters may not work one month down the line because of various issues (e.g., construction, traffic volume changes over time etc).
