  The visual medium (images and videos) naturally contains a large amount of
information redundancy, thereby providing a great opportunity for leveraging
efficiency in processing. While Vision Transformer (ViT) based models scale
effectively to large data regimes, they fail to capitalize on this inherent
redundancy, leading to higher computational costs. Mixture of Experts (MoE)
networks demonstrate scalability while maintaining same inference-time costs,
but they come with a larger parameter footprint. We present Mixture of Nested
Experts (MoNE), which utilizes a nested structure for experts, wherein
individual experts fall on an increasing compute-accuracy curve. Given a
compute budget, MoNE learns to dynamically choose tokens in a priority order,
and thus redundant tokens are processed through cheaper nested experts. Using
this framework, we achieve equivalent performance as the baseline models, while
reducing inference time compute by over two-fold. We validate our approach on
standard image and video datasets - ImageNet-21K, Kinetics400, and
Something-Something-v2. We further highlight MoNE$'$s adaptability by
showcasing its ability to maintain strong performance across different
inference-time compute budgets on videos, using only a single trained model.
