  Recent research on deep neural networks (DNNs) has primarily focused on
improving the model accuracy. Given a proper deep learning framework, it is
generally possible to increase the depth or layer width to achieve a higher
level of accuracy. However, the huge number of model parameters imposes more
computational and memory usage overhead and leads to the parameter redundancy.
In this paper, we address the parameter redundancy problem in DNNs by replacing
conventional full projections with bilinear projections. For a fully-connected
layer with $D$ input nodes and $D$ output nodes, applying bilinear projection
can reduce the model space complexity from $\mathcal{O}(D^2)$ to
$\mathcal{O}(2D)$, achieving a deep model with a sub-linear layer size.
However, structured projection has a lower freedom of degree compared to the
full projection, causing the under-fitting problem. So we simply scale up the
mapping size by increasing the number of output channels, which can keep and
even boosts the model accuracy. This makes it very parameter-efficient and
handy to deploy such deep models on mobile systems with memory limitations.
Experiments on four benchmark datasets show that applying the proposed bilinear
projection to deep neural networks can achieve even higher accuracies than
conventional full DNNs, while significantly reduces the model size.
