  The vast majority of modern consumer-grade cameras employ a rolling shutter
mechanism. In dynamic geometric computer vision applications such as visual
SLAM, the so-called rolling shutter effect therefore needs to be properly taken
into account. A dedicated relative pose solver appears to be the first problem
to solve, as it is of eminent importance to bootstrap any derivation of
multi-view geometry. However, despite its significance, it has received
inadequate attention to date.
  This paper presents a detailed investigation of the geometry of the rolling
shutter relative pose problem. We introduce the rolling shutter essential
matrix, and establish its link to existing models such as the push-broom
cameras, summarized in a clean hierarchy of multi-perspective cameras. The
generalization of well-established concepts from epipolar geometry is completed
by a definition of the Sampson distance in the rolling shutter case. The work
is concluded with a careful investigation of the introduced epipolar geometry
for rolling shutter cameras on several dedicated benchmarks.
