  Prior work in multi-task learning has mainly focused on predictions on a
single image. In this work, we present a new approach for multi-task learning
from videos via efficient inter-frame local attention (MILA). Our approach
contains a novel inter-frame attention module which allows learning of
task-specific attention across frames. We embed the attention module in a
``slow-fast'' architecture, where the slower network runs on sparsely sampled
keyframes and the light-weight shallow network runs on non-keyframes at a high
frame rate. We also propose an effective adversarial learning strategy to
encourage the slow and fast network to learn similar features. Our approach
ensures low-latency multi-task learning while maintaining high quality
predictions. Experiments show competitive accuracy compared to state-of-the-art
on two multi-task learning benchmarks while reducing the number of floating
point operations (FLOPs) by up to 70\%. In addition, our attention based
feature propagation method (ILA) outperforms prior work in terms of task
accuracy while also reducing up to 90\% of FLOPs.
