  The Concept Bottleneck Models (CBMs) of Koh et al. [2020] provide a means to
ensure that a neural network based classifier bases its predictions solely on
human understandable concepts. The concept labels, or rationales as we refer to
them, are learned by the concept labeling component of the CBM. Another
component learns to predict the target classification label from these
predicted concept labels. Unfortunately, these models are heavily reliant on
human provided concept labels for each datapoint. To enable CBMs to behave
robustly when these labels are not readily available, we show how to equip them
with the ability to abstain from predicting concepts when the concept labeling
component is uncertain. In other words, our model learns to provide rationales
for its predictions, but only whenever it is sure the rationale is correct.
