  In the field of machine learning there is a growing interest towards more
robust and generalizable algorithms. This is for example important to bridge
the gap between the environment in which the training data was collected and
the environment where the algorithm is deployed. Machine learning algorithms
have increasingly been shown to excel in finding patterns and correlations from
data. Determining the consistency of these patterns and for example the
distinction between causal correlations and nonsensical spurious relations has
proven to be much more difficult. In this paper a regularization scheme is
introduced that prefers universal causal correlations. This approach is based
on 1) the robustness of causal correlations and 2) the data not being
independently and identically distribute (i.i.d.). The scheme is demonstrated
with a classification task by clustering the (non-i.i.d.) training set in
subpopulations. A non-i.i.d. regularization term is then introduced that
penalizes weights that are not invariant over these clusters. The resulting
algorithm favours correlations that are universal over the subpopulations and
indeed a better performance is obtained on an out-of-distribution test set with
respect to a more conventional l_2-regularization.
