  Generalization of time series prediction remains an important open issue in
machine learning, wherein earlier methods have either large generalization
error or local minima. We develop an analytically solvable, unsupervised
learning scheme that extracts the most informative components for predicting
future inputs, termed predictive principal component analysis (PredPCA). Our
scheme can effectively remove unpredictable noise and minimize test prediction
error through convex optimization. Mathematical analyses demonstrate that,
provided with sufficient training samples and sufficiently high-dimensional
observations, PredPCA can asymptotically identify hidden states, system
parameters, and dimensionalities of canonical nonlinear generative processes,
with a global convergence guarantee. We demonstrate the performance of PredPCA
using sequential visual inputs comprising hand-digits, rotating 3D objects, and
natural scenes. It reliably estimates distinct hidden states and predicts
future outcomes of previously unseen test input data, based exclusively on
noisy observations. The simple architecture and low computational cost of
PredPCA are highly desirable for neuromorphic hardware.
