  Deep learning algorithms lack human-interpretable accounts of how they
transform raw visual input into a robust semantic understanding, which impedes
comparisons between different architectures, training objectives, and the human
brain. In this work, we take inspiration from neuroscience and employ
representational approaches to shed light on how neural networks encode
information at low (visual saliency) and high (semantic similarity) levels of
abstraction. Moreover, we introduce a custom image dataset where we
systematically manipulate salient and semantic information. We find that
ResNets are more sensitive to saliency information than ViTs, when trained with
object classification objectives. We uncover that networks suppress saliency in
early layers, a process enhanced by natural language supervision (CLIP) in
ResNets. CLIP also enhances semantic encoding in both architectures. Finally,
we show that semantic encoding is a key factor in aligning AI with human visual
perception, while saliency suppression is a non-brain-like strategy.
