  Over past few years afterward the birth of ResNet, skip connection has become
the defacto standard for the design of modern architectures due to its
widespread adoption, easy optimization and proven performance. Prior work has
explained the effectiveness of the skip connection mechanism from different
perspectives. In this work, we deep dive into the model's behaviors with skip
connections which can be formulated as a learnable Markov chain. An efficient
Markov chain is preferred as it always maps the input data to the target domain
in a better way. However, while a model is explained as a Markov chain, it is
not guaranteed to be optimized following an efficient Markov chain by existing
SGD-based optimizers which are prone to get trapped in local optimal points. In
order to towards a more efficient Markov chain, we propose a simple routine of
penal connection to make any residual-like model become a learnable Markov
chain. Aside from that, the penal connection can also be viewed as a particular
model regularization and can be easily implemented with one line of code in the
most popular deep learning frameworks~\footnote{Source code:
\url{https://github.com/densechen/penal-connection}}. The encouraging
experimental results in multi-modal translation and image recognition
empirically confirm our conjecture of the learnable Markov chain view and
demonstrate the superiority of the proposed penal connection.
