  Recent work on mini-batch consistency (MBC) for set functions has brought
attention to the need for sequentially processing and aggregating chunks of a
partitioned set while guaranteeing the same output for all partitions. However,
existing constraints on MBC architectures lead to models with limited
expressive power. Additionally, prior work has not addressed how to deal with
large sets during training when the full set gradient is required. To address
these issues, we propose a Universally MBC (UMBC) class of set functions which
can be used in conjunction with arbitrary non-MBC components while still
satisfying MBC, enabling a wider range of function classes to be used in MBC
settings. Furthermore, we propose an efficient MBC training algorithm which
gives an unbiased approximation of the full set gradient and has a constant
memory overhead for any set size for both train- and test-time. We conduct
extensive experiments including image completion, text classification,
unsupervised clustering, and cancer detection on high-resolution images to
verify the efficiency and efficacy of our scalable set encoding framework. Our
code is available at github.com/jeffwillette/umbc
