  In this work we study parallelization of online learning, a core primitive in
machine learning. In a parallel environment all known approaches for parallel
online learning lead to delayed updates, where the model is updated using
out-of-date information. In the worst case, or when examples are temporally
correlated, delay can have a very adverse effect on the learning algorithm.
Here, we analyze and present preliminary empirical results on a set of learning
architectures based on a feature sharding approach that present various
tradeoffs between delay, degree of parallelism, representation power and
empirical performance.
