  We consider the problem of learning good trajectories for manipulation tasks.
This is challenging because the criterion defining a good trajectory varies
with users, tasks and environments. In this paper, we propose a co-active
online learning framework for teaching robots the preferences of its users for
object manipulation tasks. The key novelty of our approach lies in the type of
feedback expected from the user: the human user does not need to demonstrate
optimal trajectories as training data, but merely needs to iteratively provide
trajectories that slightly improve over the trajectory currently proposed by
the system. We argue that this co-active preference feedback can be more easily
elicited from the user than demonstrations of optimal trajectories, which are
often challenging and non-intuitive to provide on high degrees of freedom
manipulators. Nevertheless, theoretical regret bounds of our algorithm match
the asymptotic rates of optimal trajectory algorithms. We demonstrate the
generalizability of our algorithm on a variety of grocery checkout tasks, for
whom, the preferences were not only influenced by the object being manipulated
but also by the surrounding environment.\footnote{For more details and a
demonstration video, visit: \url{http://pr.cs.cornell.edu/coactive}}
