What's the difference between probability and statistics, and why are they studied together?
The short answer to this I've heard from Persi Diaconis is the following:
The problems considered by probability and statistics are inverse to each other.  In probability theory we consider some underlying process which has some randomness or uncertainty modeled by random variables, and we figure out what happens.  In statistics we observe something that has happened, and try to figure out what underlying process would explain those observations.
