  Automated audio captioning aims at generating textual descriptions for an
audio clip. To evaluate the quality of generated audio captions, previous works
directly adopt image captioning metrics like SPICE and CIDEr, without
justifying their suitability in this new domain, which may mislead the
development of advanced models. This problem is still unstudied due to the lack
of human judgment datasets on caption quality. Therefore, we firstly construct
two evaluation benchmarks, AudioCaps-Eval and Clotho-Eval. They are established
with pairwise comparison instead of absolute rating to achieve better
inter-annotator agreement. Current metrics are found in poor correlation with
human annotations on these datasets. To overcome their limitations, we propose
a metric named FENSE, where we combine the strength of Sentence-BERT in
capturing similarity, and a novel Error Detector to penalize erroneous
sentences for robustness. On the newly established benchmarks, FENSE
outperforms current metrics by 14-25% accuracy. Code, data and web demo
available at: https://github.com/blmoistawinde/fense
