  We introduce a novel approach for prompt mixing, aiming to generate images at
the intersection of multiple text prompts using pre-trained text-to-image
diffusion models. At each time step during diffusion denoising, our algorithm
forecasts predictions w.r.t. the generated image and makes informed text
conditioning decisions. To do so, we leverage the connection between diffusion
models (rooted in non-equilibrium thermodynamics) and the Black-Scholes model
for pricing options in Finance, and draw analogies between the variables in
both contexts to derive an appropriate algorithm for prompt mixing using the
Black Scholes model. Specifically, the parallels between diffusion models and
the Black-Scholes model enable us to leverage properties related to the
dynamics of the Markovian model derived in the Black-Scholes algorithm. Our
prompt-mixing algorithm is data-efficient, meaning it does not need additional
training. Furthermore, it operates without human intervention or hyperparameter
tuning. We highlight the benefits of our approach by comparing it qualitatively
and quantitatively to other prompt mixing techniques, including linear
interpolation, alternating prompts, step-wise prompt switching, and CLIP-guided
prompt selection across various scenarios such as single object per text
prompt, multiple objects per text prompt and objects against backgrounds. Code
is available at https://github.com/divyakraman/BlackScholesDiffusion2024.
