  In recent years, significant attention in deep learning theory has been
devoted to analyzing when models that interpolate their training data can still
generalize well to unseen examples. Many insights have been gained from
studying models with multiple layers of Gaussian random features, for which one
can compute precise generalization asymptotics. However, few works have
considered the effect of weight anisotropy; most assume that the random
features are generated using independent and identically distributed Gaussian
weights, and allow only for structure in the input data. Here, we use the
replica trick from statistical physics to derive learning curves for models
with many layers of structured Gaussian features. We show that allowing
correlations between the rows of the first layer of features can aid
generalization, while structure in later layers is generally detrimental. Our
results shed light on how weight structure affects generalization in a simple
class of solvable models.
