  Quantile regression is the task of estimating a specified percentile
response, such as the median, from a collection of known covariates. We study
quantile regression with rectified linear unit (ReLU) neural networks as the
chosen model class. We derive an upper bound on the expected mean squared error
of a ReLU network used to estimate any quantile conditional on a set of
covariates. This upper bound only depends on the best possible approximation
error, the number of layers in the network, and the number of nodes per layer.
We further show upper bounds that are tight for two large classes of functions:
compositions of H\"older functions and members of a Besov space. These tight
bounds imply ReLU networks with quantile regression achieve minimax rates for
broad collections of function types. Unlike existing work, the theoretical
results hold under minimal assumptions and apply to general error
distributions, including heavy-tailed distributions. Empirical simulations on a
suite of synthetic response functions demonstrate the theoretical results
translate to practical implementations of ReLU networks. Overall, the
theoretical and empirical results provide insight into the strong performance
of ReLU neural networks for quantile regression across a broad range of
function classes and error distributions. All code for this paper is publicly
available at https://github.com/tansey/quantile-regression.
