  Deep autoregressive models are one of the most powerful models that exist
today which achieve state-of-the-art bits per dim. However, they lie at a
strict disadvantage when it comes to controlled sample generation compared to
latent variable models. Latent variable models such as VAEs and normalizing
flows allow meaningful semantic manipulations in latent space, which
autoregressive models do not have. In this paper, we propose using Fisher
scores as a method to extract embeddings from an autoregressive model to use
for interpolation and show that our method provides more meaningful sample
manipulation compared to alternate embeddings such as network activations.
