  Bounds on the smallest eigenvalue of the neural tangent kernel (NTK) are a
key ingredient in the analysis of neural network optimization and memorization.
However, existing results require distributional assumptions on the data and
are limited to a high-dimensional setting, where the input dimension $d_0$
scales at least logarithmically in the number of samples $n$. In this work we
remove both of these requirements and instead provide bounds in terms of a
measure of the collinearity of the data: notably these bounds hold with high
probability even when $d_0$ is held constant versus $n$. We prove our results
through a novel application of the hemisphere transform.
