  Real-time CNN-based object detection models for applications like
surveillance can achieve high accuracy but are computationally expensive.
Recent works have shown 10 to 100x reduction in computation cost for inference
by using domain-specific networks. However, prior works have focused on
inference only. If the domain model requires frequent retraining, training
costs can pose a significant bottleneck. To address this, we propose Dataset
Culling: a pipeline to reduce the size of the dataset for training, based on
the prediction difficulty. Images that are easy to classify are filtered out
since they contribute little to improving the accuracy. The difficulty is
measured using our proposed confidence loss metric with little computational
overhead. Dataset Culling is extended to optimize the image resolution to
further improve training and inference costs. We develop fixed-angle,
long-duration video datasets across several domains, and we show that the
dataset size can be culled by a factor of 300x to reduce the total training
time by 47x with no accuracy loss or even with slight improvement. Codes are
available: https://github.com/kentaroy47/DatasetCulling
