  Quality control is a crucial activity performed by manufacturing enterprises
to ensure that their products meet quality standards and avoid potential damage
to the brand's reputation. The decreased cost of sensors and connectivity
enabled increasing digitalization of manufacturing. In addition, artificial
intelligence enables higher degrees of automation, reducing overall costs and
time required for defect inspection. This research compares three active
learning approaches, having single and multiple oracles, to visual inspection.
Six new metrics are proposed to assess the quality of calibration without the
need for ground truth. Furthermore, this research explores whether existing
calibrators can improve their performance by leveraging an approximate ground
truth to enlarge the calibration set. The experiments were performed on
real-world data provided by Philips Consumer Lifestyle BV. Our results show
that the explored active learning settings can reduce the data labeling effort
by between three and four percent without detriment to the overall quality
goals, considering a threshold of p=0.95. Furthermore, the results show that
the proposed calibration metrics successfully capture relevant information
otherwise available to metrics used up to date only through ground truth data.
Therefore, the proposed metrics can be used to estimate the quality of models'
probability calibration without committing to a labeling effort to obtain
ground truth data.
