  Recently, optimization has become an emerging tool for neuroscientists to
study neural code. In the visual system, neurons respond to images with graded
and noisy responses. Image patterns eliciting highest responses are diagnostic
of the coding content of the neuron. To find these patterns, we have used
black-box optimizers to search a 4096d image space, leading to the evolution of
images that maximize neuronal responses. Although genetic algorithm (GA) has
been commonly used, there haven't been any systematic investigations to reveal
the best performing optimizer or the underlying principles necessary to improve
them.
  Here, we conducted a large scale in silico benchmark of optimizers for
activation maximization and found that Covariance Matrix Adaptation (CMA)
excelled in its achieved activation. We compared CMA against GA and found that
CMA surpassed the maximal activation of GA by 66% in silico and 44% in vivo. We
analyzed the structure of Evolution trajectories and found that the key to
success was not covariance matrix adaptation, but local search towards
informative dimensions and an effective step size decay. Guided by these
principles and the geometry of the image manifold, we developed SphereCMA
optimizer which competed well against CMA, proving the validity of the
identified principles. Code available at
https://github.com/Animadversio/ActMax-Optimizer-Dev
