  Recently, significant progress has been made in understanding the
generalization of neural networks (NNs) trained by gradient descent (GD) using
the algorithmic stability approach. However, most of the existing research has
focused on one-hidden-layer NNs and has not addressed the impact of different
network scaling parameters. In this paper, we greatly extend the previous work
\cite{lei2022stability,richards2021stability} by conducting a comprehensive
stability and generalization analysis of GD for multi-layer NNs. For two-layer
NNs, our results are established under general network scaling parameters,
relaxing previous conditions. In the case of three-layer NNs, our technical
contribution lies in demonstrating its nearly co-coercive property by utilizing
a novel induction strategy that thoroughly explores the effects of
over-parameterization. As a direct application of our general findings, we
derive the excess risk rate of $O(1/\sqrt{n})$ for GD algorithms in both
two-layer and three-layer NNs. This sheds light on sufficient or necessary
conditions for under-parameterized and over-parameterized NNs trained by GD to
attain the desired risk rate of $O(1/\sqrt{n})$. Moreover, we demonstrate that
as the scaling parameter increases or the network complexity decreases, less
over-parameterization is required for GD to achieve the desired error rates.
Additionally, under a low-noise condition, we obtain a fast risk rate of
$O(1/n)$ for GD in both two-layer and three-layer NNs.
