  In recent years, optimization in the learned latent space of deep generative
models has been successfully applied to black-box optimization problems such as
drug design, image generation or neural architecture search. Existing models
thereby leverage the ability of neural models to learn the data distribution
from a limited amount of samples such that new samples from the distribution
can be drawn. In this work, we propose a novel image generative approach that
optimizes the generated sample with respect to a continuously quantifiable
property. While we anticipate absolutely no practically meaningful application
for the proposed framework, it is theoretically principled and allows to
quickly propose samples at the mere boundary of the training data distribution.
Specifically, we propose to use tree-based ensemble models as mathematical
programs over the discrete latent space of vector quantized VAEs, which can be
globally solved. Subsequent weighted retraining on these queries allows to
induce a distribution shift. In lack of a practically relevant problem, we
consider a visually appealing application: the generation of happily smiling
faces (where the training distribution only contains less happy people) - and
show the principled behavior of our approach in terms of improved FID and
higher smile degree over baseline approaches.
