  Many real-world problems in machine learning, signal processing, and
communications assume that an unknown vector $x$ is measured by a matrix A,
resulting in a vector $y=Ax+z$, where $z$ denotes the noise; we call this a
single measurement vector (SMV) problem. Sometimes, multiple dependent vectors
$x^{(j)}, j\in \{1,...,J\}$, are measured at the same time, forming the
so-called multi-measurement vector (MMV) problem. Both SMV and MMV are linear
models (LM's), and the process of estimating the underlying vector(s) $x$ from
an LM given the matrices, noisy measurements, and knowledge of the noise
statistics, is called a linear inverse problem. In some scenarios, the matrix A
is stored in a single processor and this processor also records its
measurements $y$; this is called centralized LM. In other scenarios, multiple
sites are measuring the same underlying unknown vector $x$, where each site
only possesses part of the matrix A; we call this multi-processor LM. Recently,
due to an ever-increasing amount of data and ever-growing dimensions in LM's,
it has become more important to study large-scale linear inverse problems. In
this dissertation, we take advantage of tools in statistical physics and
information theory to advance the understanding of large-scale linear inverse
problems. The intuition of the application of statistical physics to our
problem is that statistical physics deals with large-scale problems, and we can
make an analogy between an LM and a thermodynamic system. In terms of
information theory, although it was originally developed to characterize the
theoretic limits of digital communication systems, information theory was later
found to be rather useful in analyzing and understanding other inference
problems. (The full abstract cannot fit in due to the space limit. Please refer
to the PDF.)
