  Unsupervised learning aims at the discovery of hidden structure that drives
the observations in the real world. It is essential for success in modern
machine learning. Latent variable models are versatile in unsupervised learning
and have applications in almost every domain. Training latent variable models
is challenging due to the non-convexity of the likelihood objective. An
alternative method is based on the spectral decomposition of low order moment
tensors. This versatile framework is guaranteed to estimate the correct model
consistently. My thesis spans both theoretical analysis of tensor decomposition
framework and practical implementation of various applications. This thesis
presents theoretical results on convergence to globally optimal solution of
tensor decomposition using the stochastic gradient descent, despite
non-convexity of the objective. This is the first work that gives global
convergence guarantees for the stochastic gradient descent on non-convex
functions with exponentially many local minima and saddle points. This thesis
also presents large-scale deployment of spectral methods carried out on various
platforms. Dimensionality reduction techniques such as random projection are
incorporated for a highly parallel and scalable tensor decomposition algorithm.
We obtain a gain in both accuracies and in running times by several orders of
magnitude compared to the state-of-art variational methods. To solve real world
problems, more advanced models and learning algorithms are proposed. This
thesis discusses generalization of LDA model to mixed membership stochastic
block model for learning user communities in social network, convolutional
dictionary model for learning word-sequence embeddings, hierarchical tensor
decomposition and latent tree structure model for learning disease hierarchy,
and spatial point process mixture model for detecting cell types in
neuroscience.
