  We consider regression estimation with modified ReLU neural networks in which
network weight matrices are first modified by a function $\alpha$ before being
multiplied by input vectors. We give an example of continuous, piecewise linear
function $\alpha$ for which the empirical risk minimizers over the classes of
modified ReLU networks with $l_1$ and squared $l_2$ penalties attain, up to a
logarithmic factor, the minimax rate of prediction of unknown $\beta$-smooth
function.
