  Most existing works focus on direct perturbations to the victim's
state/action or the underlying transition dynamics to demonstrate the
vulnerability of reinforcement learning agents to adversarial attacks. However,
such direct manipulations may not be always realizable. In this paper, we
consider a multi-agent setting where a well-trained victim agent $\nu$ is
exploited by an attacker controlling another agent $\alpha$ with an
\textit{adversarial policy}. Previous models do not account for the possibility
that the attacker may only have partial control over $\alpha$ or that the
attack may produce easily detectable "abnormal" behaviors. Furthermore, there
is a lack of provably efficient defenses against these adversarial policies. To
address these limitations, we introduce a generalized attack framework that has
the flexibility to model to what extent the adversary is able to control the
agent, and allows the attacker to regulate the state distribution shift and
produce stealthier adversarial policies. Moreover, we offer a provably
efficient defense with polynomial convergence to the most robust victim policy
through adversarial training with timescale separation. This stands in sharp
contrast to supervised learning, where adversarial training typically provides
only \textit{empirical} defenses. Using the Robosumo competition experiments,
we show that our generalized attack formulation results in much stealthier
adversarial policies when maintaining the same winning rate as baselines.
Additionally, our adversarial training approach yields stable learning dynamics
and less exploitable victim policies.
