  A deeper network structure generally handles more complicated non-linearity
and performs more competitively. Nowadays, advanced network designs often
contain a large number of repetitive structures (e.g., Transformer). They
empower the network capacity to a new level but also increase the model size
inevitably, which is unfriendly to either model restoring or transferring. In
this study, we are the first to investigate the representative potential of
fixed random weights with limited unique values by learning diverse masks and
introduce the Parameter-Efficient Masking Networks (PEMN). It also naturally
leads to a new paradigm for model compression to diminish the model size.
Concretely, motivated by the repetitive structures in modern neural networks,
we utilize one random initialized layer, accompanied with different masks, to
convey different feature mappings and represent repetitive network modules.
Therefore, the model can be expressed as \textit{one-layer} with a bunch of
masks, which significantly reduce the model storage cost. Furthermore, we
enhance our strategy by learning masks for a model filled by padding a given
random weights vector. In this way, our method can further lower the space
complexity, especially for models without many repetitive architectures. We
validate the potential of PEMN learning masks on random weights with limited
unique values and test its effectiveness for a new compression paradigm based
on different network architectures. Code is available at
https://github.com/yueb17/PEMN
