  Word-level Quality Estimation (QE) of Machine Translation (MT) aims to find
out potential translation errors in the translated sentence without reference.
Typically, conventional works on word-level QE are designed to predict the
translation quality in terms of the post-editing effort, where the word labels
("OK" and "BAD") are automatically generated by comparing words between MT
sentences and the post-edited sentences through a Translation Error Rate (TER)
toolkit. While the post-editing effort can be used to measure the translation
quality to some extent, we find it usually conflicts with the human judgement
on whether the word is well or poorly translated. To overcome the limitation,
we first create a golden benchmark dataset, namely \emph{HJQE} (Human Judgement
on Quality Estimation), where the expert translators directly annotate the
poorly translated words on their judgements. Additionally, to further make use
of the parallel corpus, we propose the self-supervised pre-training with two
tag correcting strategies, namely tag refinement strategy and tree-based
annotation strategy, to make the TER-based artificial QE corpus closer to
\emph{HJQE}. We conduct substantial experiments based on the publicly available
WMT En-De and En-Zh corpora. The results not only show our proposed dataset is
more consistent with human judgment but also confirm the effectiveness of the
proposed tag correcting strategies.\footnote{The data can be found at
\url{https://github.com/ZhenYangIACAS/HJQE}.}
