  When a model makes a consequential decision, e.g., denying someone a loan, it
needs to additionally generate actionable, realistic feedback on what the
person can do to favorably change the decision. We cast this problem through
the lens of program synthesis, in which our goal is to synthesize an optimal
(realistically cheapest or simplest) sequence of actions that if a person
executes successfully can change their classification. We present a novel and
general approach that combines search-based program synthesis and test-time
adversarial attacks to construct action sequences over a domain-specific set of
actions. We demonstrate the effectiveness of our approach on a number of deep
neural networks.
