  Test-Time Adaptation (TTA) is a critical paradigm for tackling distribution
shifts during inference, especially in visual recognition tasks. However, while
acoustic models face similar challenges due to distribution shifts in test-time
speech, TTA techniques specifically designed for acoustic modeling in the
context of open-world data shifts remain scarce. This gap is further
exacerbated when considering the unique characteristics of acoustic foundation
models: 1) they are primarily built on transformer architectures with layer
normalization and 2) they deal with test-time speech data of varying lengths in
a non-stationary manner. These aspects make the direct application of
vision-focused TTA methods, which are mostly reliant on batch normalization and
assume independent samples, infeasible. In this paper, we delve into TTA for
pre-trained acoustic models facing open-world data shifts. We find that noisy,
high-entropy speech frames, often non-silent, carry key semantic content.
Traditional TTA methods might inadvertently filter out this information using
potentially flawed heuristics. In response, we introduce a heuristic-free,
learning-based adaptation enriched by confidence enhancement. Noting that
speech signals' short-term consistency, we also apply consistency
regularization during test-time optimization. Our experiments on synthetic and
real-world datasets affirm our method's superiority over existing baselines.
