  The acoustic variability of noisy and reverberant speech mixtures is
influenced by multiple factors, such as the spectro-temporal characteristics of
the target speaker and the interfering noise, the signal-to-noise ratio (SNR)
and the room characteristics. This large variability poses a major challenge
for learning-based speech enhancement systems, since a mismatch between the
training and testing conditions can substantially reduce the performance of the
system. Generalization to unseen conditions is typically assessed by testing
the system with a new speech, noise or binaural room impulse response (BRIR)
database different from the one used during training. However, the difficulty
of the speech enhancement task can change across databases, which can
substantially influence the results. The present study introduces a
generalization assessment framework that uses a reference model trained on the
test condition, such that it can be used as a proxy for the difficulty of the
test condition. This allows to disentangle the effect of the change in task
difficulty from the effect of dealing with new data, and thus to define a new
measure of generalization performance termed the generalization gap. The
procedure is repeated in a cross-validation fashion by cycling through multiple
speech, noise, and BRIR databases to accurately estimate the generalization
gap. The proposed framework is applied to evaluate the generalization potential
of a feedforward neural network (FFNN), Conv-TasNet, DCCRN and MANNER. We find
that for all models, the performance degrades the most in speech mismatches,
while good noise and room generalization can be achieved by training on
multiple databases. Moreover, while recent models show higher performance in
matched conditions, their performance substantially decreases in mismatched
conditions and can become inferior to that of the FFNN-based system.
