  Solving multiple visual tasks using individual models can be
resource-intensive, while multi-task learning can conserve resources by sharing
knowledge across different tasks. Despite the benefits of multi-task learning,
such techniques can struggle with balancing the loss for each task, leading to
potential performance degradation. We present a novel computation- and
parameter-sharing framework that balances efficiency and accuracy to perform
multiple visual tasks utilizing individually-trained single-task transformers.
Our method is motivated by transfer learning schemes to reduce computational
and parameter storage costs while maintaining the desired performance. Our
approach involves splitting the tasks into a base task and the other sub-tasks,
and sharing a significant portion of activations and parameters/weights between
the base and sub-tasks to decrease inter-task redundancies and enhance
knowledge sharing. The evaluation conducted on NYUD-v2 and PASCAL-context
datasets shows that our method is superior to the state-of-the-art
transformer-based multi-task learning techniques with higher accuracy and
reduced computational resources. Moreover, our method is extended to video
stream inputs, further reducing computational costs by efficiently sharing
information across the temporal domain as well as the task domain. Our codes
and models will be publicly available.
