  Attention module does not always help deep models learn causal features that
are robust in any confounding context, e.g., a foreground object feature is
invariant to different backgrounds. This is because the confounders trick the
attention to capture spurious correlations that benefit the prediction when the
training and testing data are IID (identical & independent distribution); while
harm the prediction when the data are OOD (out-of-distribution). The sole
fundamental solution to learn causal attention is by causal intervention, which
requires additional annotations of the confounders, e.g., a "dog" model is
learned within "grass+dog" and "road+dog" respectively, so the "grass" and
"road" contexts will no longer confound the "dog" recognition. However, such
annotation is not only prohibitively expensive, but also inherently
problematic, as the confounders are elusive in nature. In this paper, we
propose a causal attention module (CaaM) that self-annotates the confounders in
unsupervised fashion. In particular, multiple CaaMs can be stacked and
integrated in conventional attention CNN and self-attention Vision Transformer.
In OOD settings, deep models with CaaM outperform those without it
significantly; even in IID settings, the attention localization is also
improved by CaaM, showing a great potential in applications that require robust
visual saliency. Codes are available at \url{https://github.com/Wangt-CN/CaaM}.
