  The audio spectrogram is a time-frequency representation that has been widely
used for audio classification. One of the key attributes of the audio
spectrogram is the temporal resolution, which depends on the hop size used in
the Short-Time Fourier Transform (STFT). Previous works generally assume the
hop size should be a constant value (e.g., 10 ms). However, a fixed temporal
resolution is not always optimal for different types of sound. The temporal
resolution affects not only classification accuracy but also computational
cost. This paper proposes a novel method, DiffRes, that enables differentiable
temporal resolution modeling for audio classification. Given a spectrogram
calculated with a fixed hop size, DiffRes merges non-essential time frames
while preserving important frames. DiffRes acts as a "drop-in" module between
an audio spectrogram and a classifier and can be jointly optimized with the
classification task. We evaluate DiffRes on five audio classification tasks,
using mel-spectrograms as the acoustic features, followed by off-the-shelf
classifier backbones. Compared with previous methods using the fixed temporal
resolution, the DiffRes-based method can achieve the equivalent or better
classification accuracy with at least 25% computational cost reduction. We
further show that DiffRes can improve classification accuracy by increasing the
temporal resolution of input acoustic features, without adding to the
computational cost.
