  Protecting user privacy is a major concern for many machine learning systems
that are deployed at scale and collect from a diverse set of population. One
way to address this concern is by collecting and releasing data labels in an
aggregated manner so that the information about a single user is potentially
combined with others. In this paper, we explore the possibility of training
machine learning models with aggregated data labels, rather than individual
labels. Specifically, we consider two natural aggregation procedures suggested
by practitioners: curated bags where the data points are grouped based on
common features and random bags where the data points are grouped randomly in
bag of similar sizes. For the curated bag setting and for a broad range of loss
functions, we show that we can perform gradient-based learning without any
degradation in performance that may result from aggregating data. Our method is
based on the observation that the sum of the gradients of the loss function on
individual data examples in a curated bag can be computed from the aggregate
label without the need for individual labels. For the random bag setting, we
provide a generalization risk bound based on the Rademacher complexity of the
hypothesis class and show how empirical risk minimization can be regularized to
achieve the smallest risk bound. In fact, in the random bag setting, there is a
trade-off between size of the bag and the achievable error rate as our bound
indicates. Finally, we conduct a careful empirical study to confirm our
theoretical findings. In particular, our results suggest that aggregate
learning can be an effective method for preserving user privacy while
maintaining model accuracy.
