  We propose Beat Transformer, a novel Transformer encoder architecture for
joint beat and downbeat tracking. Different from previous models that track
beats solely based on the spectrogram of an audio mixture, our model deals with
demixed spectrograms with multiple instrument channels. This is inspired by the
fact that humans perceive metrical structures from richer musical contexts,
such as chord progression and instrumentation. To this end, we develop a
Transformer model with both time-wise attention and instrument-wise attention
to capture deep-buried metrical cues. Moreover, our model adopts a novel
dilated self-attention mechanism, which achieves powerful hierarchical
modelling with only linear complexity. Experiments demonstrate a significant
improvement in demixed beat tracking over the non-demixed version. Also, Beat
Transformer achieves up to 4% point improvement in downbeat tracking accuracy
over the TCN architectures. We further discover an interpretable attention
pattern that mirrors our understanding of hierarchical metrical structures.
