  Recently, Convolutional Neural Network (CNN) has achieved excellent
performance in the classification task. It is widely known that CNN is deemed
as a 'black-box', which is hard for understanding the prediction mechanism and
debugging the wrong prediction. Some model debugging and explanation works are
developed for solving the above drawbacks. However, those methods focus on
explanation and diagnosing possible causes for model prediction, based on which
the researchers handle the following optimization of models manually. In this
paper, we propose the first completely automatic model diagnosing and treating
tool, termed as Model Doctor. Based on two discoveries that 1) each category is
only correlated with sparse and specific convolution kernels, and 2)
adversarial samples are isolated while normal samples are successive in the
feature space, a simple aggregate gradient constraint is devised for
effectively diagnosing and optimizing CNN classifiers. The aggregate gradient
strategy is a versatile module for mainstream CNN classifiers. Extensive
experiments demonstrate that the proposed Model Doctor applies to all existing
CNN classifiers, and improves the accuracy of $16$ mainstream CNN classifiers
by 1%-5%.
