  Underspecification and fairness in machine learning (ML) applications have
recently become two prominent issues in the ML community. Acoustic scene
classification (ASC) applications have so far remained unaffected by this
discussion, but are now becoming increasingly used in real-world systems where
fairness and reliability are critical aspects. In this work, we argue for the
need of a more holistic evaluation process for ASC models through disaggregated
evaluations. This entails taking into account performance differences across
several factors, such as city, location, and recording device. Although these
factors play a well-understood role in the performance of ASC models, most
works report single evaluation metrics taking into account all different strata
of a particular dataset. We argue that metrics computed on specific
sub-populations of the underlying data contain valuable information about the
expected real-world behaviour of proposed systems, and their reporting could
improve the transparency and trustability of such systems. We demonstrate the
effectiveness of the proposed evaluation process in uncovering
underspecification and fairness problems exhibited by several standard ML
architectures when trained on two widely-used ASC datasets. Our evaluation
shows that all examined architectures exhibit large biases across all factors
taken into consideration, and in particular with respect to the recording
location. Additionally, different architectures exhibit different biases even
though they are trained with the same experimental configurations.
