  We introduce and analyze an improved variant of nearest neighbors (NN) for
estimation with missing data in latent factor models. We consider a matrix
completion problem with missing data, where the $(i, t)$-th entry, when
observed, is given by its mean $f(u_i, v_t)$ plus mean-zero noise for an
unknown function $f$ and latent factors $u_i$ and $v_t$. Prior NN strategies,
like unit-unit NN, for estimating the mean $f(u_i, v_t)$ relies on existence of
other rows $j$ with $u_j \approx u_i$. Similarly, time-time NN strategy relies
on existence of columns $t'$ with $v_{t'} \approx v_t$. These strategies
provide poor performance respectively when similar rows or similar columns are
not available. Our estimate is doubly robust to this deficit in two ways: (1)
As long as there exist either good row or good column neighbors, our estimate
provides a consistent estimate. (2) Furthermore, if both good row and good
column neighbors exist, it provides a (near-)quadratic improvement in the
non-asymptotic error and admits a significantly narrower asymptotic confidence
interval when compared to both unit-unit or time-time NN.
