  This technical report describes the CP-JKU team's submission for Task 4 Sound
Event Detection with Heterogeneous Training Datasets and Potentially Missing
Labels of the DCASE 24 Challenge. We fine-tune three large Audio Spectrogram
Transformers, PaSST, BEATs, and ATST, on the joint DESED and MAESTRO datasets
in a two-stage training procedure. The first stage closely matches the baseline
system setup and trains a CRNN model while keeping the large pre-trained
transformer model frozen. In the second stage, both CRNN and transformer are
fine-tuned using heavily weighted self-supervised losses. After the second
stage, we compute strong pseudo-labels for all audio clips in the training set
using an ensemble of all three fine-tuned transformers. Then, in a second
iteration, we repeat the two-stage training process and include a distillation
loss based on the pseudo-labels, boosting single-model performance
substantially. Additionally, we pre-train PaSST and ATST on the subset of
AudioSet that comes with strong temporal labels, before fine-tuning them on the
Task 4 datasets.
