  Modeling long-term dependencies for audio signals is a particularly
challenging problem, as even small-time scales yield on the order of a hundred
thousand samples. With the recent advent of Transformers, neural architectures
became good at modeling dependencies over longer time scales, but they suffered
from quadratic constraints to scale them. We propose a generative
auto-regressive architecture that can model audio waveforms over quite a large
context, greater than 500,000 samples. Our work is adapted to learn time
dependencies by learning a latent representation by a CNN front-end, and then
learning dependencies over these representations using Transformer encoders,
fully trained end-to-end: thereby allowing to learn representations as it deems
fit for the next sample. Unlike previous works that compared different time
scales to show improvement, we use a standard dataset, with the same number of
parameters/context to show improvements. We achieve a state-of-the-art
performance as compared to other approaches such as Wavenet, SaSHMI, and
Sample-RNN on a standard dataset for modeling long-term structure. This work
gives very exciting direction for the field, given improvements in context
modeling that can be scaled with more data, as well as potentially better
results by using billions/trillions of parameters.
