  Stochastic Gradient Descent (SGD), a widely used optimization algorithm in
deep learning, is often limited to converging to local optima due to the
non-convex nature of the problem. Leveraging these local optima to improve
model performance remains a challenging task. Given the inherent complexity of
neural networks, the simple arithmetic averaging of the obtained local optima
models in undesirable results. This paper proposes a {\em soft merging} method
that facilitates rapid merging of multiple models, simplifies the merging of
specific parts of neural networks, and enhances robustness against malicious
models with extreme values. This is achieved by learning gate parameters
through a surrogate of the $l_0$ norm using hard concrete distribution without
modifying the model weights of the given local optima models. This merging
process not only enhances the model performance by converging to a better local
optimum, but also minimizes computational costs, offering an efficient and
explicit learning process integrated with stochastic gradient descent. Thorough
experiments underscore the effectiveness and superior performance of the merged
neural networks.
