  Task learning in neural networks typically requires finding a globally
optimal minimizer to a loss function objective. Conventional designs of swarm
based optimization methods apply a fixed update rule, with possibly an adaptive
step-size for gradient descent based optimization. While these methods gain
huge success in solving different optimization problems, there are some cases
where these schemes are either inefficient or suffering from local-minimum. We
present a new particle-swarm-based framework utilizing Gaussian Process
Regression to learn the underlying dynamical process of descent. The biggest
advantage of this approach is greater exploration around the current state
before deciding a descent direction. Empirical results show our approach can
escape from the local minima compare with the widely-used state-of-the-art
optimizers when solving non-convex optimization problems. We also test our
approach under high-dimensional parameter space case, namely, image
classification task.
