  While deep neural networks are highly effective at solving complex tasks,
their computational demands can hinder their usefulness in real-time
applications and with limited-resources systems. Besides, for many tasks it is
known that these models are over-parametrized: neoteric works have broadly
focused on reducing the width of these networks, rather than their depth. In
this paper, we aim to reduce the depth of over-parametrized deep neural
networks: we propose an eNtropy-basEd Pruning as a nEural Network depTH's
rEducer (NEPENTHE) to alleviate deep neural networks' computational burden.
Based on our theoretical finding, NEPENTHE focuses on un-structurally pruning
connections in layers with low entropy to remove them entirely. We validate our
approach on popular architectures such as MobileNet and Swin-T, showing that
when encountering an over-parametrization regime, it can effectively linearize
some layers (hence reducing the model's depth) with little to no performance
loss. The code will be publicly available upon acceptance of the article.
