  Binary analysis is an important capability required for many security and
software engineering applications. Consequently, there are many binary analysis
techniques and tools with varied capabilities. However, testing these tools
requires a large, varied binary dataset with corresponding source-level
information. In this paper, we present Cornucopia, an architecture agnostic
automated framework that can generate a plethora of binaries from corresponding
program source by exploiting compiler optimizations and feedback-guided
learning. Our evaluation shows that Cornucopia was able to generate 309K
binaries across four architectures (x86, x64, ARM, MIPS) with an average of 403
binaries for each program and outperforms Bintuner, a similar technique. Our
experiments revealed issues with the LLVM optimization scheduler resulting in
compiler crashes ($\sim$300). Our evaluation of four popular binary analysis
tools Angr, Ghidra, Idapro, and Radare, using Cornucopia generated binaries,
revealed various issues with these tools. Specifically, we found 263 crashes in
Angr and one memory corruption issue in Idapro. Our differential testing on the
analysis results revealed various semantic bugs in these tools. We also tested
machine learning tools, Asmvec, Safe, and Debin, that claim to capture binary
semantics and show that they perform poorly (For instance, Debin F1 score
dropped to 12.9% from reported 63.1%) on Cornucopia generated binaries. In
summary, our exhaustive evaluation shows that Cornucopia is an effective
mechanism to generate binaries for testing binary analysis techniques
effectively.
