  Early action prediction aims to successfully predict the class label of an
action before it is completely performed. This is a challenging task because
the beginning stages of different actions can be very similar, with only minor
subtle differences for discrimination. In this paper, we propose a novel Expert
Retrieval and Assembly (ERA) module that retrieves and assembles a set of
experts most specialized at using discriminative subtle differences, to
distinguish an input sample from other highly similar samples. To encourage our
model to effectively use subtle differences for early action prediction, we
push experts to discriminate exclusively between samples that are highly
similar, forcing these experts to learn to use subtle differences that exist
between those samples. Additionally, we design an effective Expert Learning
Rate Optimization method that balances the experts' optimization and leads to
better performance. We evaluate our ERA module on four public action datasets
and achieve state-of-the-art performance.
