  Autonomous robots can benefit greatly from human-provided semantic
characterizations of uncertain task environments and states. However, the
development of integrated strategies which let robots model, communicate, and
act on such 'soft data' remains challenging. Here, the Human Assisted Robotic
Planning and Sensing (HARPS) framework is presented for active semantic sensing
and planning in human-robot teams to address these gaps by formally combining
the benefits of online sampling-based POMDP policies, multimodal semantic
interaction, and Bayesian data fusion. This approach lets humans
opportunistically impose model structure and extend the range of semantic soft
data in uncertain environments by sketching and labeling arbitrary landmarks
across the environment. Dynamic updating of the environment model while during
search allows robotic agents to actively query humans for novel and relevant
semantic data, thereby improving beliefs of unknown environments and states for
improved online planning. Simulations of a UAV-enabled target search
application in a large-scale partially structured environment show significant
improvements in time and belief state estimates required for interception
versus conventional planning based solely on robotic sensing. Human subject
studies in the same environment (n = 36) demonstrate an average doubling in
dynamic target capture rate compared to the lone robot case, and highlight the
robustness of active probabilistic reasoning and semantic sensing over a range
of user characteristics and interaction modalities.
