  Traditional automatic evaluation metrics for machine translation have been
widely criticized by linguists due to their low accuracy, lack of transparency,
focus on language mechanics rather than semantics, and low agreement with human
quality evaluation. Human evaluations in the form of MQM-like scorecards have
always been carried out in real industry setting by both clients and
translation service providers (TSPs). However, traditional human translation
quality evaluations are costly to perform and go into great linguistic detail,
raise issues as to inter-rater reliability (IRR) and are not designed to
measure quality of worse than premium quality translations. In this work, we
introduce HOPE, a task-oriented and human-centric evaluation framework for
machine translation output based on professional post-editing annotations. It
contains only a limited number of commonly occurring error types, and use a
scoring model with geometric progression of error penalty points (EPPs)
reflecting error severity level to each translation unit. The initial
experimental work carried out on English-Russian language pair MT outputs on
marketing content type of text from highly technical domain reveals that our
evaluation framework is quite effective in reflecting the MT output quality
regarding both overall system-level performance and segment-level transparency,
and it increases the IRR for error type interpretation. The approach has
several key advantages, such as ability to measure and compare less than
perfect MT output from different systems, ability to indicate human perception
of quality, immediate estimation of the labor effort required to bring MT
output to premium quality, low-cost and faster application, as well as higher
IRR. Our experimental data is available at
\url{https://github.com/lHan87/HOPE}.
