  ResNet or DenseNet? Nowadays, most deep learning based approaches are
implemented with seminal backbone networks, among them the two arguably most
famous ones are ResNet and DenseNet. Despite their competitive performance and
overwhelming popularity, inherent drawbacks exist for both of them. For ResNet,
the identity shortcut that stabilizes training also limits its representation
capacity, while DenseNet has a higher capacity with multi-layer feature
concatenation. However, the dense concatenation causes a new problem of
requiring high GPU memory and more training time. Partially due to this, it is
not a trivial choice between ResNet and DenseNet. This paper provides a unified
perspective of dense summation to analyze them, which facilitates a better
understanding of their core difference. We further propose dense weighted
normalized shortcuts as a solution to the dilemma between them. Our proposed
dense shortcut inherits the design philosophy of simple design in ResNet and
DenseNet. On several benchmark datasets, the experimental results show that the
proposed DSNet achieves significantly better results than ResNet, and achieves
comparable performance as DenseNet but requiring fewer computation resources.
