  Attributed to the ever-increasing large image datasets, Convolutional Neural
Networks (CNNs) have become popular for vision-based tasks. It is generally
admirable to have larger-sized datasets for higher network training accuracies.
However, the impact of dataset quality has not to be involved. It is reasonable
to assume the near-duplicate images exist in the datasets. For instance, the
Street View House Numbers (SVHN) dataset having cropped house plate digits from
0 to 9 are likely to have repetitive digits from the same/similar house plates.
Redundant images may take up a certain portion of the dataset without
consciousness. While contributing little to no accuracy improvement for the
CNNs training, these duplicated images unnecessarily pose extra resource and
computation consumption. To this end, this paper proposes a framework to assess
the impact of the near-duplicate images on CNN training performance, called
CE-Dedup. Specifically, CE-Dedup associates a hashing-based image deduplication
approach with downstream CNNs-based image classification tasks. CE-Dedup
balances the tradeoff between a large deduplication ratio and a stable accuracy
by adjusting the deduplication threshold. The effectiveness of CE-Dedup is
validated through extensive experiments on well-known CNN benchmarks. On one
hand, while maintaining the same validation accuracy, CE-Dedup can reduce the
dataset size by 23%. On the other hand, when allowing a small validation
accuracy drop (by 5%), CE-Dedup can trim the dataset size by 75%.
