  For using neural networks in safety critical domains, it is important to know
if a decision made by a neural network is supported by prior similarities in
training. We propose runtime neuron activation pattern monitoring - after the
standard training process, one creates a monitor by feeding the training data
to the network again in order to store the neuron activation patterns in
abstract form. In operation, a classification decision over an input is further
supplemented by examining if a pattern similar (measured by Hamming distance)
to the generated pattern is contained in the monitor. If the monitor does not
contain any pattern similar to the generated pattern, it raises a warning that
the decision is not based on the training data. Our experiments show that, by
adjusting the similarity-threshold for activation patterns, the monitors can
report a significant portion of misclassfications to be not supported by
training with a small false-positive rate, when evaluated on a test set.
