  In recent decades, a number of ways of dealing with causality in practice,
such as propensity score matching, the PC algorithm and invariant causal
prediction, have been introduced. Besides its interpretational appeal, the
causal model provides the best out-of-sample prediction guarantees. In this
paper, we study the identification of causal-like models from in-sample data
that provide out-of-sample risk guarantees when predicting a target variable
from a set of covariates.
  Whereas ordinary least squares provides the best in-sample risk with limited
out-of-sample guarantees, causal models have the best out-of-sample guarantees
but achieve an inferior in-sample risk. By defining a trade-off of these
properties, we introduce $\textit{causal regularization}$. As the
regularization is increased, it provides estimators whose risk is more stable
across sub-samples at the cost of increasing their overall in-sample risk. The
increased risk stability is shown to lead to out-of-sample risk guarantees. We
provide finite sample risk bounds for all models and prove the adequacy of
cross-validation for attaining these bounds.
