  The concept bottleneck model (CBM) is an interpretable-by-design framework
that makes decisions by first predicting a set of interpretable concepts, and
then predicting the class label based on the given concepts. Existing CBMs are
trained with a fixed set of concepts (concepts are either annotated by the
dataset or queried from language models). However, this closed-world assumption
is unrealistic in practice, as users may wonder about the role of any desired
concept in decision-making after the model is deployed. Inspired by the large
success of recent vision-language pre-trained models such as CLIP in zero-shot
classification, we propose "OpenCBM" to equip the CBM with open vocabulary
concepts via: (1) Aligning the feature space of a trainable image feature
extractor with that of a CLIP's image encoder via a prototype based feature
alignment; (2) Simultaneously training an image classifier on the downstream
dataset; (3) Reconstructing the trained classification head via any set of
user-desired textual concepts encoded by CLIP's text encoder. To reveal
potentially missing concepts from users, we further propose to iteratively find
the closest concept embedding to the residual parameters during the
reconstruction until the residual is small enough. To the best of our
knowledge, our "OpenCBM" is the first CBM with concepts of open vocabularies,
providing users the unique benefit such as removing, adding, or replacing any
desired concept to explain the model's prediction even after a model is
trained. Moreover, our model significantly outperforms the previous
state-of-the-art CBM by 9% in the classification accuracy on the benchmark
dataset CUB-200-2011.
