  Multimodal pre-trained models, such as CLIP, are popular for zero-shot
classification due to their open-vocabulary flexibility and high performance.
However, vision-language models, which compute similarity scores between images
and class labels, are largely black-box, with limited interpretability, risk
for bias, and inability to discover new visual concepts not written down.
Moreover, in practical settings, the vocabulary for class names and attributes
of specialized concepts will not be known, preventing these methods from
performing well on images uncommon in large-scale vision-language datasets. To
address these limitations, we present a novel method that discovers
interpretable yet discriminative sets of attributes for visual recognition. We
introduce an evolutionary search algorithm that uses a large language model and
its in-context learning abilities to iteratively mutate a concept bottleneck of
attributes for classification. Our method produces state-of-the-art,
interpretable fine-grained classifiers. We outperform the latest baselines by
18.4% on five fine-grained iNaturalist datasets and by 22.2% on two KikiBouba
datasets, despite the baselines having access to privileged information about
class names.
