  A common practice in deep learning involves training large neural networks on
massive datasets to achieve high accuracy across various domains and tasks.
While this approach works well in many application areas, it often fails
drastically when processing data from a new modality with a significant
distribution shift from the data used to pre-train the model. This paper
focuses on adapting a large object detection model trained on RGB images to new
data extracted from IR images with a substantial modality shift. We propose
Modality Translator (ModTr) as an alternative to the common approach of
fine-tuning a large model to the new modality. ModTr adapts the IR input image
with a small transformation network trained to directly minimize the detection
loss. The original RGB model can then work on the translated inputs without any
further changes or fine-tuning to its parameters. Experimental results on
translating from IR to RGB images on two well-known datasets show that our
simple approach provides detectors that perform comparably or better than
standard fine-tuning, without forgetting the knowledge of the original model.
This opens the door to a more flexible and efficient service-based detection
pipeline, where a unique and unaltered server, such as an RGB detector, runs
constantly while being queried by different modalities, such as IR with the
corresponding translations model. Our code is available at:
https://github.com/heitorrapela/ModTr.
