  Residual learning has recently surfaced as an effective means of constructing
very deep neural networks for object recognition. However, current incarnations
of residual networks do not allow for the modeling and integration of complex
relations between closely coupled recognition tasks or across domains. Such
problems are often encountered in multimedia applications involving large-scale
content recognition. We propose a novel extension of residual learning for deep
networks that enables intuitive learning across multiple related tasks using
cross-connections called cross-residuals. These cross-residuals connections can
be viewed as a form of in-network regularization and enables greater network
generalization. We show how cross-residual learning (CRL) can be integrated in
multitask networks to jointly train and detect visual concepts across several
tasks. We present a single multitask cross-residual network with >40% less
parameters that is able to achieve competitive, or even better, detection
performance on a visual sentiment concept detection problem normally requiring
multiple specialized single-task networks. The resulting multitask
cross-residual network also achieves better detection performance by about
10.4% over a standard multitask residual network without cross-residuals with
even a small amount of cross-task weighting.
