  Increasing a ML model accuracy is not enough, we must also increase its
trustworthiness. This is an important step for building resilient AI systems
for safety-critical applications such as automotive, finance, and healthcare.
For that purpose, we propose a multi-agent system that combines both machine
and human agents. In this system, a checker agent calculates a trust score of
each instance (which penalizes overconfidence and overcautiousness in
predictions) using an agreement-based method and ranks it; then an improver
agent filters the anomalous instances based on a human rule-based procedure
(which is considered safe), gets the human labels, applies geometric data
augmentation, and retrains with the augmented data using transfer learning. We
evaluate the system on corrupted versions of the MNIST and FashionMNIST
datasets. We get an improvement in accuracy and trust score with just few
additional labels compared to a baseline approach.
