  Efficient deep neural network (DNN) models equipped with compact operators
(e.g., depthwise convolutions) have shown great potential in reducing DNNs'
theoretical complexity (e.g., the total number of weights/operations) while
maintaining a decent model accuracy. However, existing efficient DNNs are still
limited in fulfilling their promise in boosting real-hardware efficiency, due
to their commonly adopted compact operators' low hardware utilization. In this
work, we open up a new compression paradigm for developing real-hardware
efficient DNNs, leading to boosted hardware efficiency while maintaining model
accuracy. Interestingly, we observe that while some DNN layers' activation
functions help DNNs' training optimization and achievable accuracy, they can be
properly removed after training without compromising the model accuracy.
Inspired by this observation, we propose a framework dubbed DepthShrinker,
which develops hardware-friendly compact networks via shrinking the basic
building blocks of existing efficient DNNs that feature irregular computation
patterns into dense ones with much improved hardware utilization and thus
real-hardware efficiency. Excitingly, our DepthShrinker framework delivers
hardware-friendly compact networks that outperform both state-of-the-art
efficient DNNs and compression techniques, e.g., a 3.06% higher accuracy and
1.53$\times$ throughput on Tesla V100 over SOTA channel-wise pruning method
MetaPruning. Our codes are available at:
https://github.com/facebookresearch/DepthShrinker.
