  In this work, we address music representation learning using convolution-free
transformers. We build on top of existing spectrogram-based audio transformers
such as AST and train our models on a supervised task using patchout training
similar to PaSST. In contrast to previous works, we study how specific design
decisions affect downstream music tagging tasks instead of focusing on the
training task. We assess the impact of initializing the models with different
pre-trained weights, using various input audio segment lengths, using learned
representations from different blocks and tokens of the transformer for
downstream tasks, and applying patchout at inference to speed up feature
extraction. We find that 1) initializing the model from ImageNet or AudioSet
weights and using longer input segments are beneficial both for the training
and downstream tasks, 2) the best representations for the considered downstream
tasks are located in the middle blocks of the transformer, and 3) using
patchout at inference allows faster processing than our convolutional baselines
while maintaining superior performance. The resulting models, MAEST, are
publicly available and obtain the best performance among open models in music
tagging tasks.
