  One of the most common problems preventing the application of prediction
models in the real world is lack of generalization: The accuracy of models,
measured in the benchmark does repeat itself on future data, e.g. in the
settings of real business. There is relatively little methods exist that
estimate the confidence of prediction models. In this paper, we propose novel
methods that, given a neural network classification model, estimate uncertainty
of particular predictions generated by this model. Furthermore, we propose a
method that, given a model and a confidence level, calculates a threshold that
separates prediction generated by this model into two subsets, one of them
meets the given confidence level. In contrast to other methods, the proposed
methods do not require any changes on existing neural networks, because they
simply build on the output logit layer of a common neural network. In
particular, the methods infer the confidence of a particular prediction based
on the distribution of the logit values corresponding to this prediction. The
proposed methods constitute a tool that is recommended for filtering
predictions in the process of knowledge extraction, e.g. based on web
scrapping, where predictions subsets are identified that maximize the precision
on cost of the recall, which is less important due to the availability of data.
The method has been tested on different tasks including relation extraction,
named entity recognition and image classification to show the significant
increase of accuracy achieved.
