  Large-scale pre-trained models have been known that they are transferable,
and they generalize well on the unseen dataset. Recently, multimodal
pre-trained models such as CLIP show significant performance improvement in
diverse experiments. However, when the labeled dataset is limited, the
generalization of a new dataset or domain is still challenging. To improve the
generalization performance on few-shot learning, there have been diverse
efforts, such as prompt learning and adapter. However, the current few-shot
adaptation methods are not interpretable, and they require a high computation
cost for adaptation. In this study, we propose a new method, robust prompt
learning with knowledge graph (RPLKG). Based on the knowledge graph, we
automatically design diverse interpretable and meaningful prompt sets. Our
model obtains cached embeddings of prompt sets after one forwarding from a
large pre-trained model. After that, model optimizes the prompt selection
processes with GumbelSoftmax. In this way, our model is trained using
relatively little memory and learning time. Also, RPLKG selects the optimal
interpretable prompt automatically, depending on the dataset. In summary, RPLKG
is i) interpretable, ii) requires small computation resources, and iii) easy to
incorporate prior human knowledge. To validate the RPLKG, we provide
comprehensive experimental results on few-shot learning, domain generalization
and new class generalization setting. RPLKG shows a significant performance
improvement compared to zero-shot learning and competitive performance against
several prompt learning methods using much lower resources.
