  Representation learning plays a crucial role in automated feature selection,
particularly in the context of high-dimensional data, where non-parametric
methods often struggle. In this study, we focus on supervised learning
scenarios where the pertinent information resides within a lower-dimensional
linear subspace of the data, namely the multi-index model. If this subspace
were known, it would greatly enhance prediction, computation, and
interpretation. To address this challenge, we propose a novel method for joint
linear feature learning and non-parametric function estimation, aimed at more
effectively leveraging hidden features for learning. Our approach employs
empirical risk minimisation, augmented with a penalty on function derivatives,
ensuring versatility. Leveraging the orthogonality and rotation invariance
properties of Hermite polynomials, we introduce our estimator, named RegFeaL.
By using alternative minimisation, we iteratively rotate the data to improve
alignment with leading directions. We establish that the expected risk of our
method converges in high-probability to the minimal risk under minimal
assumptions and with explicit rates. Additionally, we provide empirical results
demonstrating the performance of RegFeaL in various experiments.
