  The multiplicative structure of parameters and input data in the first layer
of neural networks is explored to build connection between the landscape of the
loss function with respect to parameters and the landscape of the model
function with respect to input data. By this connection, it is shown that flat
minima regularize the gradient of the model function, which explains the good
generalization performance of flat minima. Then, we go beyond the flatness and
consider high-order moments of the gradient noise, and show that Stochastic
Gradient Descent (SGD) tends to impose constraints on these moments by a linear
stability analysis of SGD around global minima. Together with the
multiplicative structure, we identify the Sobolev regularization effect of SGD,
i.e. SGD regularizes the Sobolev seminorms of the model function with respect
to the input data. Finally, bounds for generalization error and adversarial
robustness are provided for solutions found by SGD under assumptions of the
data distribution.
