  This paper presents a reactive planning system that enriches the topological
representation of an environment with a tightly integrated semantic
representation, achieved by incorporating and exploiting advances in deep
perceptual learning and probabilistic semantic reasoning. Our architecture
combines object detection with semantic SLAM, affording robust, reactive
logical as well as geometric planning in unexplored environments. Moreover, by
incorporating a human mesh estimation algorithm, our system is capable of
reacting and responding in real time to semantically labeled human motions and
gestures. New formal results allow tracking of suitably non-adversarial moving
targets, while maintaining the same collision avoidance guarantees. We suggest
the empirical utility of the proposed control architecture with a numerical
study including comparisons with a state-of-the-art dynamic replanning
algorithm, and physical implementation on both a wheeled and legged platform in
different settings with both geometric and semantic goals.
