  This article presents a comprehensive survey of online test-time adaptation
(OTTA), focusing on effectively adapting machine learning models to
distributionally different target data upon batch arrival. Despite the recent
proliferation of OTTA methods, conclusions from previous studies are
inconsistent due to ambiguous settings, outdated backbones, and inconsistent
hyperparameter tuning, which obscure core challenges and hinder
reproducibility. To enhance clarity and enable rigorous comparison, we classify
OTTA techniques into three primary categories and benchmark them using a modern
backbone, the Vision Transformer (ViT). Our benchmarks cover conventional
corrupted datasets such as CIFAR-10/100-C and ImageNet-C, as well as real-world
shifts represented by CIFAR-10.1, OfficeHome, and CIFAR-10-Warehouse. The
CIFAR-10-Warehouse dataset includes a variety of variations from different
search engines and synthesized data generated through diffusion models. To
measure efficiency in online scenarios, we introduce novel evaluation metrics,
including GFLOPs, wall clock time, and GPU memory usage, providing a clearer
picture of the trade-offs between adaptation accuracy and computational
overhead. Our findings diverge from existing literature, revealing that (1)
transformers demonstrate heightened resilience to diverse domain shifts, (2)
the efficacy of many OTTA methods relies on large batch sizes, and (3)
stability in optimization and resistance to perturbations are crucial during
adaptation, particularly when the batch size is 1. Based on these insights, we
highlight promising directions for future research. Our benchmarking toolkit
and source code are available at https://github.com/Jo-wang/OTTA_ViT_survey.
