  One of the problems on the way to successful implementation of neural
networks is the quality of annotation. For instance, different annotators can
annotate images in a different way and very often their decisions do not match
exactly and in extreme cases are even mutually exclusive which results in noisy
annotations and, consequently, inaccurate predictions.
  To avoid that problem in the task of computed tomography (CT) imaging
segmentation we propose a clearing algorithm for annotations. It consists of 3
stages:
  - annotators scoring, which assigns a higher confidence level to better
annotators;
  - nodules scoring, which assigns a higher confidence level to nodules
confirmed by good annotators;
  - nodules merging, which aggregates annotations according to nodules
confidence.
  In general, the algorithm can be applied to many different tasks (namely,
binary and multi-class semantic segmentation, and also with trivial adjustments
to classification and regression) where there are several annotators labeling
each image.
