  Annually, research teams spend large amounts of money to evaluate the quality
of machine translation systems (WMT, inter alia). This is expensive because it
requires detailed human labor. The recently proposed annotation protocol, Error
Span Annotation (ESA), has annotators marking erroneous parts of the
translation. In our work, we help the annotators by pre-filling the span
annotations with automatic quality estimation. With AI assistance, we obtain
more detailed annotations while cutting down the time per span annotation by
half (71s/error span $\rightarrow$ 31s/error span). The biggest advantage of
ESA$^\mathrm{AI}$ protocol is an accurate priming of annotators (pre-filled
error spans) before they assign the final score as opposed to starting from
scratch. In addition, the annotation budget can be reduced by up to 24% with
filtering of examples that the AI deems to be very likely to be correct.
