  The outstanding performance of large foundational models across diverse
tasks-from computer vision to speech and natural language processing-has
significantly increased their demand. However, storing and transmitting these
models pose significant challenges due to their massive size (e.g., 350GB for
GPT-3). Recent literature has focused on compressing the original weights or
reducing the number of parameters required for fine-tuning these models. These
compression methods typically involve constraining the parameter space, for
example, through low-rank reparametrization (e.g., LoRA) or quantization (e.g.,
QLoRA) during model training. In this paper, we present MCNC as a novel model
compression method that constrains the parameter space to low-dimensional
pre-defined and frozen nonlinear manifolds, which effectively cover this space.
Given the prevalence of good solutions in over-parameterized deep neural
networks, we show that by constraining the parameter space to our proposed
manifold, we can identify high-quality solutions while achieving unprecedented
compression rates across a wide variety of tasks. Through extensive experiments
in computer vision and natural language processing tasks, we demonstrate that
our method, MCNC, significantly outperforms state-of-the-art baselines in terms
of compression, accuracy, and/or model reconstruction time.
