  Retrieval networks are essential for searching and indexing. Compared to
classification networks, attention visualization for retrieval networks is
hardly studied. We formulate attention visualization as a constrained
optimization problem. We leverage the unit L2-Norm constraint as an attention
filter (L2-CAF) to localize attention in both classification and retrieval
networks. Unlike recent literature, our approach requires neither architectural
changes nor fine-tuning. Thus, a pre-trained network's performance is never
undermined
  L2-CAF is quantitatively evaluated using weakly supervised object
localization. State-of-the-art results are achieved on classification networks.
For retrieval networks, significant improvement margins are achieved over a
Grad-CAM baseline. Qualitative evaluation demonstrates how the L2-CAF
visualizes attention per frame for a recurrent retrieval network. Further
ablation studies highlight the computational cost of our approach and compare
L2-CAF with other feasible alternatives. Code available at
https://bit.ly/3iDBLFv
