  Robust estimation of the essential matrix, which encodes the relative
position and orientation of two cameras, is a fundamental step in structure
from motion pipelines. Recent deep-based methods achieved accurate estimation
by using complex network architectures that involve graphs, attention layers,
and hard pruning steps. Here, we propose a simpler network architecture based
on Deep Sets. Given a collection of point matches extracted from two images,
our method identifies outlier point matches and models the displacement noise
in inlier matches. A weighted DLT module uses these predictions to regress the
essential matrix. Our network achieves accurate recovery that is superior to
existing networks with significantly more complex architectures.
