  Test-time adaptation has proven effective in adapting a given trained model
to unseen test samples with potential distribution shifts. However, in
real-world scenarios, models are usually deployed on resource-limited devices,
e.g., FPGAs, and are often quantized and hard-coded with non-modifiable
parameters for acceleration. In light of this, existing methods are often
infeasible since they heavily depend on computation-intensive backpropagation
for model updating that may be not supported. To address this, we propose a
test-time Forward-Optimization Adaptation (FOA) method. In FOA, we seek to
solely learn a newly added prompt (as model's input) via a derivative-free
covariance matrix adaptation evolution strategy. To make this strategy work
stably under our online unsupervised setting, we devise a novel fitness
function by measuring test-training statistic discrepancy and model prediction
entropy. Moreover, we design an activation shifting scheme that directly tunes
the model activations for shifted test samples, making them align with the
source training domain, thereby further enhancing adaptation performance.
Without using any backpropagation and altering model weights, FOA runs on
quantized 8-bit ViT outperforms gradient-based TENT on full-precision 32-bit
ViT, while achieving an up to 24-fold memory reduction on ImageNet-C.
