  Although deep neural networks are well-known for their remarkable performance
in tackling complex tasks, their hunger for computational resources remains a
significant hurdle, posing energy-consumption issues and restricting their
deployment on resource-constrained devices, which stalls their widespread
adoption. In this paper, we present an optimal transport method to reduce the
depth of over-parametrized deep neural networks, alleviating their
computational burden. More specifically, we propose a new regularization
strategy based on the Max-Sliced Wasserstein distance to minimize the distance
between the intermediate feature distributions in the neural network. We show
that minimizing this distance enables the complete removal of intermediate
layers in the network, with almost no performance loss and without requiring
any finetuning. We assess the effectiveness of our method on traditional image
classification setups. We commit to releasing the source code upon acceptance
of the article.
