  Binary code analysis plays a pivotal role in various software security
applications, such as software maintenance, malware detection, software
vulnerability discovery, patch analysis, etc. However, unlike source code,
understanding binary code is challenging for reverse engineers due to the
absence of semantic information. Therefore, automated tools are needed to
assist human players in interpreting binary code. In recent years, two groups
of technologies have shown promising prospects: (1) Deep learning-based
technologies have demonstrated competitive results in tasks related to binary
code understanding, furthermore, (2) Large Language Models (LLMs) have been
extensively pre-trained at the source-code level for tasks such as code
understanding and generation. This makes participants wonder about the ability
of LLMs in binary code understanding.
  In this work, we propose a benchmark to evaluate the effectiveness of LLMs in
real-world reverse engineering scenarios. The benchmark covers two key binary
code understanding tasks, including function name recovery and binary code
summarization. We gain valuable insights into their capabilities and
limitations through extensive evaluations of popular LLMs using our benchmark.
Our evaluations reveal that existing LLMs can understand binary code to a
certain extent, thereby improving the efficiency of binary code analysis. Our
results highlight the great potential of the LLMs in advancing the field of
binary code understanding.
