  Despite being highly performant, deep neural networks might base their
decisions on features that spuriously correlate with the provided labels, thus
hurting generalization. To mitigate this, 'model guidance' has recently gained
popularity, i.e. the idea of regularizing the models' explanations to ensure
that they are "right for the right reasons". While various techniques to
achieve such model guidance have been proposed, experimental validation of
these approaches has thus far been limited to relatively simple and / or
synthetic datasets. To better understand the effectiveness of the various
design choices that have been explored in the context of model guidance, in
this work we conduct an in-depth evaluation across various loss functions,
attribution methods, models, and 'guidance depths' on the PASCAL VOC 2007 and
MS COCO 2014 datasets. As annotation costs for model guidance can limit its
applicability, we also place a particular focus on efficiency. Specifically, we
guide the models via bounding box annotations, which are much cheaper to obtain
than the commonly used segmentation masks, and evaluate the robustness of model
guidance under limited (e.g. with only 1% of annotated images) or overly coarse
annotations. Further, we propose using the EPG score as an additional
evaluation metric and loss function ('Energy loss'). We show that optimizing
for the Energy loss leads to models that exhibit a distinct focus on
object-specific features, despite only using bounding box annotations that also
include background regions. Lastly, we show that such model guidance can
improve generalization under distribution shifts. Code available at:
https://github.com/sukrutrao/Model-Guidance.
