  This paper investigates the impact of multiscale data on machine learning
algorithms, particularly in the context of deep learning. A dataset is
multiscale if its distribution shows large variations in scale across different
directions. This paper reveals multiscale structures in the loss landscape,
including its gradients and Hessians inherited from the data. Correspondingly,
it introduces a novel gradient descent approach, drawing inspiration from
multiscale algorithms used in scientific computing. This approach seeks to
transcend empirical learning rate selection, offering a more systematic,
data-informed strategy to enhance training efficiency, especially in the later
stages.
