  While Reinforcement Learning ( RL) has made great strides towards solving
increasingly complicated problems, many algorithms are still brittle to even
slight environmental changes. Contextual Reinforcement Learning (cRL) provides
a framework to model such changes in a principled manner, thereby enabling
flexible, precise and interpretable task specification and generation. Our goal
is to show how the framework of cRL contributes to improving zero-shot
generalization in RL through meaningful benchmarks and structured reasoning
about generalization tasks. We confirm the insight that optimal behavior in cRL
requires context information, as in other related areas of partial
observability. To empirically validate this in the cRL framework, we provide
various context-extended versions of common RL environments. They are part of
the first benchmark library, CARL, designed for generalization based on cRL
extensions of popular benchmarks, which we propose as a testbed to further
study general agents. We show that in the contextual setting, even simple RL
environments become challenging - and that naive solutions are not enough to
generalize across complex context spaces.
