  Given the ever-increasing computational costs of modern machine learning
models, we need to find new ways to reuse such expert models and thus tap into
the resources that have been invested in their creation. Recent work suggests
that the power of these massive models is captured by the representations they
learn. Therefore, we seek a model that can relate between different existing
representations and propose to solve this task with a conditionally invertible
network. This network demonstrates its capability by (i) providing generic
transfer between diverse domains, (ii) enabling controlled content synthesis by
allowing modification in other domains, and (iii) facilitating diagnosis of
existing representations by translating them into interpretable domains such as
images. Our domain transfer network can translate between fixed representations
without having to learn or finetune them. This allows users to utilize various
existing domain-specific expert models from the literature that had been
trained with extensive computational resources. Experiments on diverse
conditional image synthesis tasks, competitive image modification results and
experiments on image-to-image and text-to-image generation demonstrate the
generic applicability of our approach. For example, we translate between BERT
and BigGAN, state-of-the-art text and image models to provide text-to-image
generation, which neither of both experts can perform on their own.
