  Singular value decomposition (SVD) is one of the most popular compression
methods that approximate a target matrix with smaller matrices. However,
standard SVD treats the parameters within the matrix with equal importance,
which is a simple but unrealistic assumption. The parameters of a trained
neural network model may affect task performance unevenly, which suggests
non-equal importance among the parameters. Compared to SVD, the decomposition
method aware of parameter importance is the more practical choice in real
cases. Unlike standard SVD, weighted value decomposition is a non-convex
optimization problem that lacks a closed-form solution. We systematically
investigated multiple optimization strategies to tackle the problem and
examined our method by compressing Transformer-based language models. Further,
we designed a metric to predict when the SVD may introduce a significant
performance drop, for which our method can be a rescue strategy. The extensive
evaluations demonstrate that our method can perform better than current SOTA
methods in compressing Transformer-based language models.
