  The quality of acoustic echo cancellers (AECs) in real-time communication
systems is typically evaluated using objective metrics like ERLE and PESQ, and
less commonly with lab-based subjective tests like ITU-T Rec. P.831. We will
show that these objective measures are not well correlated to subjective
measures. We then introduce an open-source crowdsourcing approach for
subjective evaluation of echo impairment which can be used to evaluate the
performance of AECs. We provide a study that shows this tool is accurate and
highly reproducible. This new tool has been recently used in the ICASSP 2021
AEC Challenge which made the challenge possible to do quickly and cost
effectively.
