  In this work, we begin to investigate the possibility of training a deep
neural network on the task of binary code understanding. Specifically, the
network would take, as input, features derived directly from binaries and
output English descriptions of functionality to aid a reverse engineer in
investigating the capabilities of a piece of closed-source software, be it
malicious or benign. Given recent success in applying large language models
(generative AI) to the task of source code summarization, this seems a
promising direction. However, in our initial survey of the available datasets,
we found nothing of sufficiently high quality and volume to train these complex
models. Instead, we build our own dataset derived from a capture of Stack
Overflow containing 1.1M entries. A major result of our work is a novel dataset
evaluation method using the correlation between two distances on sample pairs:
one distance in the embedding space of inputs and the other in the embedding
space of outputs. Intuitively, if two samples have inputs close in the input
embedding space, their outputs should also be close in the output embedding
space. We found this Embedding Distance Correlation (EDC) test to be highly
diagnostic, indicating that our collected dataset and several existing
open-source datasets are of low quality as the distances are not well
correlated. We proceed to explore the general applicability of EDC, applying it
to a number of qualitatively known good datasets and a number of synthetically
known bad ones and found it to be a reliable indicator of dataset value.
