  We present a novel algorithm (Principal Sensitivity Analysis; PSA) to analyze
the knowledge of the classifier obtained from supervised machine learning
techniques. In particular, we define principal sensitivity map (PSM) as the
direction on the input space to which the trained classifier is most sensitive,
and use analogously defined k-th PSM to define a basis for the input space. We
train neural networks with artificial data and real data, and apply the
algorithm to the obtained supervised classifiers. We then visualize the PSMs to
demonstrate the PSA's ability to decompose the knowledge acquired by the
trained classifiers.
