  Since compiler optimization is the most common source contributing to binary
code differences in syntax, testing the resilience against the changes caused
by different compiler optimization settings has become a standard evaluation
step for most binary diffing approaches. For example, 47 top-venue papers in
the last 12 years compared different program versions compiled by default
optimization levels (e.g., -Ox in GCC and LLVM). Although many of them claim
they are immune to compiler transformations, it is yet unclear about their
resistance to non-default optimization settings. Especially, we have observed
that adversaries explored non-default compiler settings to amplify malware
differences.
  This paper takes the first step to systematically studying the effectiveness
of compiler optimization on binary code differences. We tailor search-based
iterative compilation for the auto-tuning of binary code differences. We
develop BinTuner to search near-optimal optimization sequences that can
maximize the amount of binary code differences. We run BinTuner with GCC 10.2
and LLVM 11.0 on SPEC benchmarks (CPU2006 & CPU2017), Coreutils, and OpenSSL.
Our experiments show that at the cost of 279 to 1,881 compilation iterations,
BinTuner can find custom optimization sequences that are substantially better
than the general -Ox settings. BinTuner's outputs seriously undermine prominent
binary diffing tools' comparisons. In addition, the detection rate of the IoT
malware variants tuned by BinTuner falls by more than 50%. Our findings paint a
cautionary tale for security analysts that attackers have a new way to mutate
malware code cost-effectively, and the research community needs to step back to
reassess optimization-resistance evaluations.
