Situationally Aware Options
  Hierarchical abstractions, also known as options -- a type of temporally
extended action (Sutton et. al. 1999) that enables a reinforcement learning
agent to plan at a higher level, abstracting away from the lower-level details.
In this work, we learn reusable options whose parameters can vary, encouraging
different behaviors, based on the current situation. In principle, these
behaviors can include vigor, defence or even risk-averseness. These are some
examples of what we refer to in the broader context as Situational Awareness
(SA). We incorporate SA, in the form of vigor, into hierarchical RL by defining
and learning situationally aware options in a Probabilistic Goal Semi-Markov
Decision Process (PG-SMDP). This is achieved using our Situationally Aware
oPtions (SAP) policy gradient algorithm which comes with a theoretical
convergence guarantee. We learn reusable options in different scenarios in a
RoboCup soccer domain (i.e., winning/losing). These options learn to execute
with different levels of vigor resulting in human-like behaviours such as
`time-wasting' in the winning scenario. We show the potential of the agent to
exit bad local optima using reusable options in RoboCup. Finally, using SAP,
the agent mitigates feature-based model misspecification in a Bottomless Pit of
Death domain.
