  Real time outdoor navigation in highly dynamic environments is an crucial
problem. The recent literature on real time static SLAM don't scale up to
dynamic outdoor environments. Most of these methods assume moving objects as
outliers or discard the information provided by them. We propose an algorithm
to jointly infer the camera trajectory and the moving object trajectory
simultaneously. In this paper, we perform a sparse scene flow based motion
segmentation using a stereo camera. The segmented objects motion models are
used for accurate localization of the camera trajectory as well as the moving
objects. We exploit the relationship between moving objects for improving the
accuracy of the poses. We formulate the poses as a factor graph incorporating
all the constraints. We achieve exact incremental solution by solving a full
nonlinear optimization problem in real time. The evaluation is performed on the
challenging KITTI dataset with multiple moving cars.Our method outperforms the
previous baselines in outdoor navigation.
