  Planes are generally used in 3D reconstruction for depth sensors, such as
RGB-D cameras and LiDARs. This paper focuses on the problem of estimating the
optimal planes and sensor poses to minimize the point-to-plane distance. The
resulting least-squares problem is referred to as plane adjustment (PA) in the
literature, which is the counterpart of bundle adjustment (BA) in visual
reconstruction. Iterative methods are adopted to solve these least-squares
problems. Typically, Newton's method is rarely used for a large-scale
least-squares problem, due to the high computational complexity of the Hessian
matrix. Instead, methods using an approximation of the Hessian matrix, such as
the Levenberg-Marquardt (LM) method, are generally adopted. This paper
challenges this ingrained idea. We adopt the Newton's method to efficiently
solve the PA problem. Specifically, given poses, the optimal planes have
close-form solutions. Thus we can eliminate planes from the cost function,
which significantly reduces the number of variables. Furthermore, as the
optimal planes are functions of poses, this method actually ensures that the
optimal planes for the current estimated poses can be obtained at each
iteration, which benefits the convergence. The difficulty lies in how to
efficiently compute the Hessian matrix and the gradient of the resulting cost.
This paper provides an efficient solution. Empirical evaluation shows that our
algorithm converges significantly faster than the widely used LM algorithm.
