  AI powered code-recommendation systems, such as Copilot and CodeWhisperer,
provide code suggestions inside a programmer's environment (e.g., an IDE) with
the aim of improving productivity. We pursue mechanisms for leveraging signals
about programmers' acceptance and rejection of code suggestions to guide
recommendations. We harness data drawn from interactions with GitHub Copilot, a
system used by millions of programmers, to develop interventions that can save
time for programmers. We introduce a utility-theoretic framework to drive
decisions about suggestions to display versus withhold. The approach,
conditional suggestion display from human feedback (CDHF), relies on a cascade
of models that provide the likelihood that recommended code will be accepted.
These likelihoods are used to selectively hide suggestions, reducing both
latency and programmer verification time. Using data from 535 programmers, we
perform a retrospective evaluation of CDHF and show that we can avoid
displaying a significant fraction of suggestions that would have been rejected.
We further demonstrate the importance of incorporating the programmer's latent
unobserved state in decisions about when to display suggestions through an
ablation study. Finally, we showcase how using suggestion acceptance as a
reward signal for guiding the display of suggestions can lead to suggestions of
reduced quality, indicating an unexpected pitfall.
