  Recent works on personalized text-to-image generation usually learn to bind a
special token with specific subjects or styles of a few given images by tuning
its embedding through gradient descent. It is natural to question whether we
can optimize the textual inversions by only accessing the process of model
inference. As only requiring the forward computation to determine the textual
inversion retains the benefits of less GPU memory, simple deployment, and
secure access for scalable models. In this paper, we introduce a
\emph{gradient-free} framework to optimize the continuous textual inversion in
an iterative evolutionary strategy. Specifically, we first initialize an
appropriate token embedding for textual inversion with the consideration of
visual and text vocabulary information. Then, we decompose the optimization of
evolutionary strategy into dimension reduction of searching space and
non-convex gradient-free optimization in subspace, which significantly
accelerates the optimization process with negligible performance loss.
Experiments in several applications demonstrate that the performance of
text-to-image model equipped with our proposed gradient-free method is
comparable to that of gradient-based counterparts with variant GPU/CPU
platforms, flexible employment, as well as computational efficiency.
