  Robot policies need to adapt to human preferences and/or new environments.
Human experts may have the domain knowledge required to help robots achieve
this adaptation. However, existing works often require costly offline
re-training on human feedback, and those feedback usually need to be frequent
and too complex for the humans to reliably provide. To avoid placing undue
burden on human experts and allow quick adaptation in critical real-world
situations, we propose designing and sparingly presenting easy-to-answer
pairwise action preference queries in an online fashion. Our approach designs
queries and determines when to present them to maximize the expected value
derived from the queries' information. We demonstrate our approach with
experiments in simulation, human user studies, and real robot experiments. In
these settings, our approach outperforms baseline techniques while presenting
fewer queries to human experts. Experiment videos, code and appendices are
found at https://sites.google.com/view/onlineactivepreferences.
