  We consider the problem of learning high-level controls over the global
structure of generated sequences, particularly in the context of symbolic music
generation with complex language models. In this work, we present the
Transformer autoencoder, which aggregates encodings of the input data across
time to obtain a global representation of style from a given performance. We
show it is possible to combine this global representation with other temporally
distributed embeddings, enabling improved control over the separate aspects of
performance style and melody. Empirically, we demonstrate the effectiveness of
our method on various music generation tasks on the MAESTRO dataset and a
YouTube dataset with 10,000+ hours of piano performances, where we achieve
improvements in terms of log-likelihood and mean listening scores as compared
to baselines.
