  Although reinforcement learning has seen remarkable progress over the last
years, solving robust dexterous object-manipulation tasks in multi-object
settings remains a challenge. In this paper, we focus on models that can learn
manipulation tasks in fixed multi-object settings and extrapolate this skill
zero-shot without any drop in performance when the number of objects changes.
We consider the generic task of bringing a specific cube out of a set to a goal
position. We find that previous approaches, which primarily leverage attention
and graph neural network-based architectures, do not generalize their skills
when the number of input objects changes while scaling as $K^2$. We propose an
alternative plug-and-play module based on relational inductive biases to
overcome these limitations. Besides exceeding performances in their training
environment, we show that our approach, which scales linearly in $K$, allows
agents to extrapolate and generalize zero-shot to any new object number.
