  The remarkable success of modern machine learning models on large datasets
often demands extensive training time and resource consumption. To save cost, a
prevalent research line, known as online batch selection, explores selecting
informative subsets during the training process. Although recent efforts
achieve advancements by measuring the impact of each sample on generalization,
their reliance on additional reference models inherently limits their practical
applications, when there are no such ideal models available. On the other hand,
the vanilla reference-model-free methods involve independently scoring and
selecting data in a sample-wise manner, which sacrifices the diversity and
induces the redundancy. To tackle this dilemma, we propose Diversified Batch
Selection (DivBS), which is reference-model-free and can efficiently select
diverse and representative samples. Specifically, we define a novel selection
objective that measures the group-wise orthogonalized representativeness to
combat the redundancy issue of previous sample-wise criteria, and provide a
principled selection-efficient realization. Extensive experiments across
various tasks demonstrate the significant superiority of DivBS in the
performance-speedup trade-off. The code is publicly available.
