  In order to better understand feature learning in neural networks, we propose
a framework for understanding linear models in tangent feature space where the
features are allowed to be transformed during training. We consider linear
transformations of features, resulting in a joint optimization over parameters
and transformations with a bilinear interpolation constraint. We show that this
optimization problem has an equivalent linearly constrained optimization with
structured regularization that encourages approximately low rank solutions.
Specializing to neural network structure, we gain insights into how the
features and thus the kernel function change, providing additional nuance to
the phenomenon of kernel alignment when the target function is poorly
represented using tangent features. We verify our theoretical observations in
the kernel alignment of real neural networks.
