  In recent years, deep neural network exhibits its powerful superiority on
information discrimination in many computer vision applications. However, the
capacity of deep neural network architecture is still a mystery to the
researchers. Intuitively, larger capacity of neural network can always deposit
more information to improve the discrimination ability of the model. But, the
learnable parameter scale is not feasible to estimate the capacity of deep
neural network. Due to the overfitting, directly increasing hidden nodes number
and hidden layer number are already demonstrated not necessary to effectively
increase the network discrimination ability.
  In this paper, we propose a novel measurement, named "total valid bits", to
evaluate the capacity of deep neural networks for exploring how to
quantitatively understand the deep learning and the insights behind its super
performance. Specifically, our scheme to retrieve the total valid bits
incorporates the skilled techniques in both training phase and inference phase.
In the network training, we design decimal weight regularization and 8-bit
forward quantization to obtain the integer-oriented network representations.
Moreover, we develop adaptive-bitwidth and non-uniform quantization strategy in
the inference phase to find the neural network capacity, total valid bits. By
allowing zero bitwidth, our adaptive-bitwidth quantization can execute the
model reduction and valid bits finding simultaneously. In our extensive
experiments, we first demonstrate that our total valid bits is a good indicator
of neural network capacity. We also analyze the impact on network capacity from
the network architecture and advanced training skills, such as dropout and
batch normalization.
