  One of the grand challenges of reinforcement learning is the ability to
generalize to new tasks. However, general agents require a set of rich, diverse
tasks to train on. Designing a `foundation environment' for such tasks is
tricky -- the ideal environment would support a range of emergent phenomena, an
expressive task space, and fast runtime. To take a step towards addressing this
research bottleneck, this work presents Powderworld, a lightweight yet
expressive simulation environment running directly on the GPU. Within
Powderworld, two motivating challenges distributions are presented, one for
world-modelling and one for reinforcement learning. Each contains hand-designed
test tasks to examine generalization. Experiments indicate that increasing the
environment's complexity improves generalization for world models and certain
reinforcement learning agents, yet may inhibit learning in high-variance
environments. Powderworld aims to support the study of generalization by
providing a source of diverse tasks arising from the same core rules.
