  Variational Inference (VI) offers a method for approximating intractable
likelihoods. In neural VI, inference of approximate posteriors is commonly done
using an encoder. Alternatively, encoderless VI offers a framework for learning
generative models from data without encountering suboptimalities caused by
amortization via an encoder (e.g. in presence of missing or uncertain data).
However, in absence of an encoder, such methods often suffer in convergence due
to the slow nature of gradient steps required to learn the approximate
posterior parameters. In this paper, we introduce Relay VI (RVI), a framework
that dramatically improves both the convergence and performance of encoderless
VI. In our experiments over multiple datasets, we study the effectiveness of
RVI in terms of convergence speed, loss, representation power and missing data
imputation. We find RVI to be a unique tool, often superior in both performance
and convergence speed to previously proposed encoderless as well as amortized
VI models (e.g. VAE).
