  Nearly all modern software suffers from bloat that negatively impacts its
performance and security. To combat this problem, several automated techniques
have been proposed to debloat software. A key metric used in many of these
works to demonstrate improved security is code reuse gadget count reduction.
The use of this metric is based on the prevailing idea that reducing the number
of gadgets available in a software package reduces its attack surface and makes
mounting a gadget-based code reuse exploit such as return-oriented programming
(ROP) more difficult for an attacker. In this paper, we challenge this idea and
show through a variety of realistic debloating scenarios the flaws inherent to
the gadget count reduction metric. Specifically, we demonstrate that software
debloating can achieve high gadget count reduction rates, yet fail to limit an
attacker's ability to construct an exploit. Worse yet, in some scenarios high
gadget count reduction rates conceal instances in which software debloating
makes security worse by introducing new, useful gadgets. To address these
issues, we propose a set of four new metrics for measuring security
improvements realized through software debloating that are quality-oriented
rather than quantity-oriented. We show that these metrics can identify when
debloating negatively impacts security and be efficiently calculated using our
static binary analysis tool, the Gadget Set Analyzer. Finally, we demonstrate
the utility of these metrics in two realistic case studies: iterative
debloating and debloater evaluation.
