  In this paper, we present a novel end-to-end network architecture to estimate
fundamental matrix directly from stereo images. To establish a complete working
pipeline, different deep neural networks in charge of finding correspondences
in images, performing outlier rejection and calculating fundamental matrix, are
integrated into an end-to-end network architecture.
  To well train the network and preserve geometry properties of fundamental
matrix, a new loss function is introduced. To evaluate the accuracy of
estimated fundamental matrix more reasonably, we design a new evaluation metric
which is highly consistent with visualization result. Experiments conducted on
both outdoor and indoor data-sets show that this network outperforms
traditional methods as well as previous deep learning based methods on various
metrics and achieves significant performance improvements.
