  This paper targets a novel trade-off problem in generalizable prompt learning
for vision-language models (VLM), i.e., improving the performance on unseen
classes while maintaining the performance on seen classes. Comparing with
existing generalizable methods that neglect the seen classes degradation, the
setting of this problem is more strict and fits more closely with practical
applications. To solve this problem, we start from the optimization
perspective, and leverage the relationship between loss landscape geometry and
model generalization ability. By analyzing the loss landscapes of the
state-of-the-art method and vanilla Sharpness-aware Minimization (SAM) based
method, we conclude that the trade-off performance correlates to both loss
value and loss sharpness, while each of them is indispensable. However, we find
the optimizing gradient of existing methods cannot maintain high relevance to
both loss value and loss sharpness during optimization, which severely affects
their trade-off performance. To this end, we propose a novel SAM-based method
for prompt learning, denoted as Gradient Constrained Sharpness-aware Context
Optimization (GCSCoOp), to dynamically constrain the optimizing gradient, thus
achieving above two-fold optimization objective simultaneously. Extensive
experiments verify the effectiveness of GCSCoOp in the trade-off problem.
