  Graph matching involves combinatorial optimization based on edge-to-edge
affinity matrix, which can be generally formulated as Lawler's Quadratic
Assignment Problem (QAP). This paper presents a QAP network directly learning
with the affinity matrix (equivalently the association graph) whereby the
matching problem is translated into a constrained vertex classification task.
The association graph is learned by an embedding network for vertex
classification, followed by Sinkhorn normalization and a cross-entropy loss for
end-to-end learning. We further improve the embedding model on association
graph by introducing Sinkhorn based matching-aware constraint, as well as dummy
nodes to deal with unequal sizes of graphs. To our best knowledge, this is one
of the first network to directly learn with the general Lawler's QAP. In
contrast, recent deep matching methods focus on the learning of node/edge
features in two graphs respectively. We also show how to extend our network to
hypergraph matching, and matching of multiple graphs. Experimental results on
both synthetic graphs and real-world images show its effectiveness. For pure
QAP tasks on synthetic data and QAPLIB benchmark, our method can perform
competitively and even surpass state-of-the-art graph matching and QAP solvers
with notable less time cost. We provide a project homepage at
http://thinklab.sjtu.edu.cn/project/NGM/index.html.
