  Capsule networks (CapsNets) are capable of modeling visual hierarchical
relationships, which is achieved by the "routing-by-agreement" mechanism. This
paper proposes a pairwise agreement mechanism to build capsules, inspired by
the feature interactions of factorization machines (FMs). The proposed method
has a much lower computation complexity. We further proposed a new CapsNet
architecture that combines the strengths of residual networks in representing
low-level visual features and CapsNets in modeling the relationships of parts
to wholes. We conduct comprehensive experiments to compare the routing
algorithms, including dynamic routing, EM routing, and our proposed FM
agreement, based on both architectures of original CapsNet and our proposed
one, and the results show that our method achieves both excellent performance
and efficiency under a variety of situations.
