  Data shuffling of training data among different computing nodes (workers) has
been identified as a core element to improve the statistical performance of
modern large-scale machine learning algorithms. Data shuffling is often
considered as one of the most significant bottlenecks in such systems due to
the heavy communication load. Under a master-worker architecture (where a
master has access to the entire dataset and only communication between the
master and the workers is allowed) coding has been recently proved to
considerably reduce the communication load. This work considers a different
communication paradigm referred to as decentralized data shuffling, where
workers are allowed to communicate with one another via a shared link. The
decentralized data shuffling problem has two phases: workers communicate with
each other during the data shuffling phase, and then workers update their
stored content during the storage phase. The main challenge is to derive novel
converse bounds and achievable schemes for decentralized data shuffling by
considering the asymmetry of the workers' storages (i.e., workers are
constrained to store different files in their storages based on the problem
setting), in order to characterize the fundamental limits of this problem. For
the case of uncoded storage (i.e., each worker directly stores a subset of bits
of the dataset), this paper proposes converse and achievable bounds (based on
distributed interference alignment and distributed clique-covering strategies)
that are within a factor of 3/2 of one another. The proposed schemes are also
exactly optimal under the constraint of uncoded storage for either large
storage size or at most four workers in the system.
