  RGB-T tracking, a vital downstream task of object tracking, has made
remarkable progress in recent years. Yet, it remains hindered by two major
challenges: 1) the trade-off between performance and efficiency; 2) the
scarcity of training data. To address the latter challenge, some recent methods
employ prompts to fine-tune pre-trained RGB tracking models and leverage
upstream knowledge in a parameter-efficient manner. However, these methods
inadequately explore modality-independent patterns and disregard the dynamic
reliability of different modalities in open scenarios. We propose M3PT, a novel
RGB-T prompt tracking method that leverages middle fusion and multi-modal and
multi-stage visual prompts to overcome these challenges. We pioneer the use of
the adjustable middle fusion meta-framework for RGB-T tracking, which could
help the tracker balance the performance with efficiency, to meet various
demands of application. Furthermore, based on the meta-framework, we utilize
multiple flexible prompt strategies to adapt the pre-trained model to
comprehensive exploration of uni-modal patterns and improved modeling of
fusion-modal features in diverse modality-priority scenarios, harnessing the
potential of prompt learning in RGB-T tracking. Evaluating on 6 existing
challenging benchmarks, our method surpasses previous state-of-the-art prompt
fine-tuning methods while maintaining great competitiveness against excellent
full-parameter fine-tuning methods, with only 0.34M fine-tuned parameters.
