  Dataset distillation synthesizes a small set of images from a large-scale
real dataset such that synthetic and real images share similar behavioral
properties (e.g, distributions of gradients or features) during a training
process. Through extensive analyses on current methods and real datasets,
together with empirical observations, we provide in this paper two important
things to share for dataset distillation. First, object parts that appear on
one side of a real image are highly likely to appear on the opposite side of
another image within a dataset, which we call the bilateral equivalence.
Second, the bilateral equivalence enforces synthetic images to duplicate
discriminative parts of objects on both the left and right sides of the images,
limiting the recognition of subtle differences between objects. To address this
problem, we introduce a surprisingly simple yet effective technique for dataset
distillation, dubbed FYI, that enables distilling rich semantics of real images
into synthetic ones. To this end, FYI embeds a horizontal flipping technique
into distillation processes, mitigating the influence of the bilateral
equivalence, while capturing more details of objects. Experiments on
CIFAR-10/100, Tiny-ImageNet, and ImageNet demonstrate that FYI can be
seamlessly integrated into several state-of-the-art methods, without modifying
training objectives and network architectures, and it improves the performance
remarkably.
