  Black-box unsupervised domain adaptation (UDA) learns with source predictions
of target data without accessing either source data or source models during
training, and it has clear superiority in data privacy and flexibility in
target network selection. However, the source predictions of target data are
often noisy and training with them is prone to learning collapses. We propose
BiMem, a bi-directional memorization mechanism that learns to remember useful
and representative information to correct noisy pseudo labels on the fly,
leading to robust black-box UDA that can generalize across different visual
recognition tasks. BiMem constructs three types of memory, including sensory
memory, short-term memory, and long-term memory, which interact in a
bi-directional manner for comprehensive and robust memorization of learnt
features. It includes a forward memorization flow that identifies and stores
useful features and a backward calibration flow that rectifies features' pseudo
labels progressively. Extensive experiments show that BiMem achieves superior
domain adaptation performance consistently across various visual recognition
tasks such as image classification, semantic segmentation and object detection.
