  Generative models have been successfully applied to image style transfer and
domain translation. However, there is still a wide gap in the quality of
results when learning such tasks on musical audio. Furthermore, most
translation models only enable one-to-one or one-to-many transfer by relying on
separate encoders or decoders and complex, computationally-heavy models. In
this paper, we introduce the Modulated Variational auto-Encoders (MoVE) to
perform musical timbre transfer. We define timbre transfer as applying parts of
the auditory properties of a musical instrument onto another. First, we show
that we can achieve this task by conditioning existing domain translation
techniques with Feature-wise Linear Modulation (FiLM). Then, we alleviate the
need for additional adversarial networks by replacing the usual translation
criterion by a Maximum Mean Discrepancy (MMD) objective. This allows a faster
and more stable training along with a controllable latent space encoder. By
further conditioning our system on several different instruments, we can
generalize to many-to-many transfer within a single variational architecture
able to perform multi-domain transfers. Our models map inputs to 3-dimensional
representations, successfully translating timbre from one instrument to another
and supporting sound synthesis from a reduced set of control parameters. We
evaluate our method in reconstruction and generation tasks while analyzing the
auditory descriptor distributions across transferred domains. We show that this
architecture allows for generative controls in multi-domain transfer, yet
remaining light, fast to train and effective on small datasets.
