  Humans have perfected the art of learning from multiple modalities through
sensory organs. Despite their impressive predictive performance on a single
modality, neural networks cannot reach human level accuracy with respect to
multiple modalities. This is a particularly challenging task due to variations
in the structure of respective modalities. Conditional Batch Normalization
(CBN) is a popular method that was proposed to learn contextual features to aid
deep learning tasks. This technique uses auxiliary data to improve
representational power by learning affine transformations for convolutional
neural networks. Despite the boost in performance observed by using CBN layers,
our work reveals that the visual features learned by introducing auxiliary data
via CBN deteriorates. We perform comprehensive experiments to evaluate the
brittleness of CBN networks to various datasets, suggesting that learning from
visual features alone could often be superior for generalization. We evaluate
CBN models on natural images for bird classification and histology images for
cancer type classification. We observe that the CBN network learns close to no
visual features on the bird classification dataset and partial visual features
on the histology dataset. Our extensive experiments reveal that CBN may
encourage shortcut learning between the auxiliary data and labels.
