  Relational reasoning refers to the ability to infer and understand the
relations between multiple entities. In humans, this ability underpins many
higher cognitive functions, such as problem solving and decision-making, and
has been reliably linked to fluid intelligence. Despite machine learning models
making impressive advances across various domains, such as natural language
processing and vision, the extent to which such models can perform relational
reasoning tasks remains unclear. Here we study the importance of positional
encoding (PE) for relational reasoning in the Transformer, and find that a
learnable PE outperforms all other commonly-used PEs (e.g., absolute, relative,
rotary, etc.). Moreover, we find that when using a PE with a learnable
parameter, the choice of initialization greatly influences the learned
representations and its downstream generalization performance. Specifically, we
find that a learned PE initialized from a small-norm distribution can 1)
uncover ground-truth position information, 2) generalize in the presence of
noisy inputs, and 3) produce behavioral patterns that are consistent with human
performance. Our results shed light on the importance of learning
high-performing and robust PEs during relational reasoning tasks, which will
prove useful for tasks in which ground truth positions are not provided or not
known.
