  It is well known that neural networks with rectified linear units (ReLU)
activation functions are positively scale-invariant. Conventional algorithms
like stochastic gradient descent optimize the neural networks in the vector
space of weights, which is, however, not positively scale-invariant. This
mismatch may lead to problems during the optimization process. Then, a natural
question is: \emph{can we construct a new vector space that is positively
scale-invariant and sufficient to represent ReLU neural networks so as to
better facilitate the optimization process }? In this paper, we provide our
positive answer to this question. First, we conduct a formal study on the
positive scaling operators which forms a transformation group, denoted as
$\mathcal{G}$. We show that the value of a path (i.e. the product of the
weights along the path) in the neural network is invariant to positive scaling
and prove that the value vector of all the paths is sufficient to represent the
neural networks under mild conditions. Second, we show that one can identify
some basis paths out of all the paths and prove that the linear span of their
value vectors (denoted as $\mathcal{G}$-space) is an invariant space with lower
dimension under the positive scaling group. Finally, we design stochastic
gradient descent algorithm in $\mathcal{G}$-space (abbreviated as
$\mathcal{G}$-SGD) to optimize the value vector of the basis paths of neural
networks with little extra cost by leveraging back-propagation. Our experiments
show that $\mathcal{G}$-SGD significantly outperforms the conventional SGD
algorithm in optimizing ReLU networks on benchmark datasets.
