  AI-based code generators are an emerging solution for automatically writing
programs starting from descriptions in natural language, by using deep neural
networks (Neural Machine Translation, NMT). In particular, code generators have
been used for ethical hacking and offensive security testing by generating
proof-of-concept attacks. Unfortunately, the evaluation of code generators
still faces several issues. The current practice uses output similarity
metrics, i.e., automatic metrics that compute the textual similarity of
generated code with ground-truth references. However, it is not clear what
metric to use, and which metric is most suitable for specific contexts. This
work analyzes a large set of output similarity metrics on offensive code
generators. We apply the metrics on two state-of-the-art NMT models using two
datasets containing offensive assembly and Python code with their descriptions
in the English language. We compare the estimates from the automatic metrics
with human evaluation and provide practical insights into their strengths and
limitations.
