  Computer vision models suffer from a phenomenon known as catastrophic
forgetting when learning novel concepts from continuously shifting training
data. Typical solutions for this continual learning problem require extensive
rehearsal of previously seen data, which increases memory costs and may violate
data privacy. Recently, the emergence of large-scale pre-trained vision
transformer models has enabled prompting approaches as an alternative to
data-rehearsal. These approaches rely on a key-query mechanism to generate
prompts and have been found to be highly resistant to catastrophic forgetting
in the well-established rehearsal-free continual learning setting. However, the
key mechanism of these methods is not trained end-to-end with the task
sequence. Our experiments show that this leads to a reduction in their
plasticity, hence sacrificing new task accuracy, and inability to benefit from
expanded parameter capacity. We instead propose to learn a set of prompt
components which are assembled with input-conditioned weights to produce
input-conditioned prompts, resulting in a novel attention-based end-to-end
key-query scheme. Our experiments show that we outperform the current SOTA
method DualPrompt on established benchmarks by as much as 4.5% in average final
accuracy. We also outperform the state of art by as much as 4.4% accuracy on a
continual learning benchmark which contains both class-incremental and
domain-incremental task shifts, corresponding to many practical settings. Our
code is available at https://github.com/GT-RIPL/CODA-Prompt
