  Semantic analysis on visible (RGB) and infrared (IR) images has gained
attention for its ability to be more accurate and robust under low-illumination
and complex weather conditions. Due to the lack of pre-trained foundation
models on the large-scale infrared image datasets, existing methods prefer to
design task-specific frameworks and directly fine-tune them with pre-trained
foundation models on their RGB-IR semantic relevance datasets, which results in
poor scalability and limited generalization. In this work, we propose a
scalable and efficient framework called UniRGB-IR to unify RGB-IR downstream
tasks, in which a novel adapter is developed to efficiently introduce richer
RGB-IR features into the pre-trained RGB-based foundation model. Specifically,
our framework consists of a vision transformer (ViT) foundation model, a
Multi-modal Feature Pool (MFP) module and a Supplementary Feature Injector
(SFI) module. The MFP and SFI modules cooperate with each other as an adpater
to effectively complement the ViT features with the contextual multi-scale
features. During training process, we freeze the entire foundation model to
inherit prior knowledge and only optimize the MFP and SFI modules. Furthermore,
to verify the effectiveness of our framework, we utilize the ViT-Base as the
pre-trained foundation model to perform extensive experiments. Experimental
results on various RGB-IR downstream tasks demonstrate that our method can
achieve state-of-the-art performance. The source code and results are available
at https://github.com/PoTsui99/UniRGB-IR.git.
