  Building machines capable of efficiently collaborating with humans has been a
longstanding goal in artificial intelligence. Especially in the presence of
uncertainties, optimal cooperation often requires that humans and artificial
agents model each other's behavior and use these models to infer underlying
goals, beliefs or intentions, potentially involving multiple levels of
recursion. Empirical evidence for such higher-order cognition in human behavior
is also provided by previous works in cognitive science, linguistics, and
robotics. We advocate for a new paradigm for active learning for human feedback
that utilises humans as active data sources while accounting for their higher
levels of agency. In particular, we discuss how increasing level of agency
results in qualitatively different forms of rational communication between an
active learning system and a teacher. Additionally, we provide a practical
example of active learning using a higher-order cognitive model. This is
accompanied by a computational study that underscores the unique behaviors that
this model produces.
