  In this paper, we aim to address the challenging task of semantic matching
where matching ambiguity is difficult to resolve even with learned deep
features. We tackle this problem by taking into account the confidence in
predictions and develop a novel refinement strategy to correct partial matching
errors. Specifically, we introduce a Confidence-Aware Semantic Matching Network
(CAMNet) which instantiates two key ideas of our approach. First, we propose to
estimate a dense confidence map for a matching prediction through
self-supervised learning. Second, based on the estimated confidence, we refine
initial predictions by propagating reliable matching to the rest of locations
on the image plane. In addition, we develop a new hybrid loss in which we
integrate a semantic alignment loss with a confidence loss, and an adversarial
loss that measures the quality of semantic correspondence. We are the first
that exploit confidence during refinement to improve semantic matching accuracy
and develop an end-to-end self-supervised adversarial learning procedure for
the entire matching network. We evaluate our method on two public benchmarks,
on which we achieve top performance over the prior state of the art. We will
release our source code at https://github.com/ShuaiyiHuang/CAMNet.
