  It is a well-known fact that the performance of deep learning models
deteriorates when they encounter a distribution shift at test time. Test-time
adaptation (TTA) algorithms have been proposed to adapt the model online while
inferring test data. However, existing research predominantly focuses on
classification tasks through the optimization of batch normalization layers or
classification heads, but this approach limits its applicability to various
model architectures like Transformers and makes it challenging to apply to
other tasks, such as object detection. In this paper, we propose a novel online
adaption approach for object detection in continually changing test domains,
considering which part of the model to update, how to update it, and when to
perform the update. By introducing architecture-agnostic and lightweight
adaptor modules and only updating these while leaving the pre-trained backbone
unchanged, we can rapidly adapt to new test domains in an efficient way and
prevent catastrophic forgetting. Furthermore, we present a practical and
straightforward class-wise feature aligning method for object detection to
resolve domain shifts. Additionally, we enhance efficiency by determining when
the model is sufficiently adapted or when additional adaptation is needed due
to changes in the test distribution. Our approach surpasses baselines on widely
used benchmarks, achieving improvements of up to 4.9\%p and 7.9\%p in mAP for
COCO $\rightarrow$ COCO-corrupted and SHIFT, respectively, while maintaining
about 20 FPS or higher.
