  Covert planning refers to a class of constrained planning problems where an
agent aims to accomplish a task with minimal information leaked to a passive
observer to avoid detection. However, existing methods of covert planning often
consider deterministic environments or do not exploit the observer's imperfect
information. This paper studies how covert planning can leverage the coupling
of stochastic dynamics and the observer's imperfect observation to achieve
optimal task performance without being detected. Specifically, we employ a
Markov decision process to model the interaction between the agent and its
stochastic environment, and a partial observation function to capture the
leaked information to a passive observer. Assuming the observer employs
hypothesis testing to detect if the observation deviates from a nominal policy,
the covert planning agent aims to maximize the total discounted reward while
keeping the probability of being detected as an adversary below a given
threshold. We prove that finite-memory policies are more powerful than
Markovian policies in covert planning. Then, we develop a primal-dual proximal
policy gradient method with a two-time-scale update to compute a (locally)
optimal covert policy. We demonstrate the effectiveness of our methods using a
stochastic gridworld example. Our experimental results illustrate that the
proposed method computes a policy that maximizes the adversary's expected
reward without violating the detection constraint, and empirically demonstrates
how the environmental noises can influence the performance of the covert
policies.
