  Incorporating relational reasoning into neural networks has greatly expanded
their capabilities and scope. One defining trait of relational reasoning is
that it operates on a set of entities, as opposed to standard vector
representations. Existing end-to-end approaches typically extract entities from
inputs by directly interpreting the latent feature representations as a set. We
show that these approaches do not respect set permutational invariance and thus
have fundamental representational limitations. To resolve this limitation, we
propose a simple and general network module called a Set Refiner Network (SRN).
We first use synthetic image experiments to demonstrate how our approach
effectively decomposes objects without explicit supervision. Then, we insert
our module into existing relational reasoning models and show that respecting
set invariance leads to substantial gains in prediction performance and
robustness on several relational reasoning tasks.
