  Continual learning aims to refine model parameters for new tasks while
retaining knowledge from previous tasks. Recently, prompt-based learning has
emerged to leverage pre-trained models to be prompted to learn subsequent tasks
without the reliance on the rehearsal buffer. Although this approach has
demonstrated outstanding results, existing methods depend on preceding
task-selection process to choose appropriate prompts. However, imperfectness in
task-selection may lead to negative impacts on the performance particularly in
the scenarios where the number of tasks is large or task distributions are
imbalanced. To address this issue, we introduce I-Prompt, a task-agnostic
approach focuses on the visual semantic information of image tokens to
eliminate task prediction. Our method consists of semantic prompt matching,
which determines prompts based on similarities between tokens, and image
token-level prompting, which applies prompts directly to image tokens in the
intermediate layers. Consequently, our method achieves competitive performance
on four benchmarks while significantly reducing training time compared to
state-of-the-art methods. Moreover, we demonstrate the superiority of our
method across various scenarios through extensive experiments.
