  While machine learning systems show high success rate in many complex tasks,
research shows they can also fail in very unexpected situations. Rise of
machine learning products in safety-critical industries cause an increase in
attention in evaluating model robustness and estimating failure probability in
machine learning systems. In this work, we propose a design to train a student
model -- a failure predictor -- to predict the main model's error for input
instances based on their saliency map. We implement and review the preliminary
results of our failure predictor model on an autonomous vehicle steering
control system as an example of safety-critical applications.
