  One key challenge for multi-task Reinforcement learning (RL) in practice is
the absence of task indicators. Robust RL has been applied to deal with task
ambiguity, but may result in over-conservative policies. To balance the
worst-case (robustness) and average performance, we propose Group
Distributionally Robust Markov Decision Process (GDR-MDP), a flexible
hierarchical MDP formulation that encodes task groups via a latent mixture
model. GDR-MDP identifies the optimal policy that maximizes the expected return
under the worst-possible qualified belief over task groups within an ambiguity
set. We rigorously show that GDR-MDP's hierarchical structure improves
distributional robustness by adding regularization to the worst possible
outcomes. We then develop deep RL algorithms for GDR-MDP for both value-based
and policy-based RL methods. Extensive experiments on Box2D control tasks,
MuJoCo benchmarks, and Google football platforms show that our algorithms
outperform classic robust training algorithms across diverse environments in
terms of robustness under belief uncertainties. Demos are available on our
project page (\url{https://sites.google.com/view/gdr-rl/home}).
