  Learning from data has led to substantial advances in a multitude of
disciplines, including text and multimedia search, speech recognition, and
autonomous-vehicle navigation. Can machine learning enable similar leaps in the
natural and social sciences? This is certainly the expectation in many
scientific fields and recent years have seen a plethora of applications of
non-linear models to a wide range of datasets. However, flexible non-linear
solutions will not always improve upon manually adding transforms and
interactions between variables to linear regression models. We discuss how to
recognize this before constructing a data-driven model and how such analysis
can help us move to intrinsically interpretable regression models. Furthermore,
for a variety of applications in the natural and social sciences we demonstrate
why improvements may be seen with more complex regression models and why they
may not.
