  Matrix and tensor completion aim to recover a low-rank matrix / tensor from
limited observations and have been commonly used in applications such as
recommender systems and multi-relational data mining. A state-of-the-art matrix
completion algorithm is Soft-Impute, which exploits the special "sparse plus
low-rank" structure of the matrix iterates to allow efficient SVD in each
iteration. Though Soft-Impute is a proximal algorithm, it is generally believed
that acceleration destroys the special structure and is thus not useful. In
this paper, we show that Soft-Impute can indeed be accelerated without
comprising this structure. To further reduce the iteration time complexity, we
propose an approximate singular value thresholding scheme based on the power
method. Theoretical analysis shows that the proposed algorithm still enjoys the
fast $O(1/T^2)$ convergence rate of accelerated proximal algorithms. We further
extend the proposed algorithm to tensor completion with the scaled latent
nuclear norm regularizer. We show that a similar "sparse plus low-rank"
structure also exists, leading to low iteration complexity and fast $O(1/T^2)$
convergence rate. Extensive experiments demonstrate that the proposed algorithm
is much faster than Soft-Impute and other state-of-the-art matrix and tensor
completion algorithms.
