  Most set prediction models in deep learning use set-equivariant operations,
but they actually operate on multisets. We show that set-equivariant functions
cannot represent certain functions on multisets, so we introduce the more
appropriate notion of multiset-equivariance. We identify that the existing Deep
Set Prediction Network (DSPN) can be multiset-equivariant without being
hindered by set-equivariance and improve it with approximate implicit
differentiation, allowing for better optimization while being faster and saving
memory. In a range of toy experiments, we show that the perspective of
multiset-equivariance is beneficial and that our changes to DSPN achieve better
results in most cases. On CLEVR object property prediction, we substantially
improve over the state-of-the-art Slot Attention from 8% to 77% in one of the
strictest evaluation metrics because of the benefits made possible by implicit
differentiation.
