  We investigate whether prompts learned independently for different tasks can
be later combined through prompt algebra to obtain a model that supports
composition of tasks. We consider Visual Language Models (VLM) with prompt
tuning as our base classifier and formally define the notion of prompt algebra.
We propose constrained prompt tuning to improve performance of the composite
classifier. In the proposed scheme, prompts are constrained to appear in the
lower dimensional subspace spanned by the basis vectors of the pre-trained
vocabulary. Further regularization is added to ensure that the learned prompt
is grounded correctly to the existing pre-trained vocabulary. We demonstrate
the effectiveness of our method on object classification and object-attribute
classification datasets. On average, our composite model obtains classification
accuracy within 2.5% of the best base model. On UTZappos it improves
classification accuracy over the best base model by 8.45% on average.
