  In this paper, we focus on multi-task classification, where related
classification tasks share the same label space and are learned simultaneously.
In particular, we tackle a new setting, which is more realistic than currently
addressed in the literature, where categories shift from training to test data.
Hence, individual tasks do not contain complete training data for the
categories in the test set. To generalize to such test data, it is crucial for
individual tasks to leverage knowledge from related tasks. To this end, we
propose learning an association graph to transfer knowledge among tasks for
missing classes. We construct the association graph with nodes representing
tasks, classes and instances, and encode the relationships among the nodes in
the edges to guide their mutual knowledge transfer. By message passing on the
association graph, our model enhances the categorical information of each
instance, making it more discriminative. To avoid spurious correlations between
task and class nodes in the graph, we introduce an assignment entropy
maximization that encourages each class node to balance its edge weights. This
enables all tasks to fully utilize the categorical information from related
tasks. An extensive evaluation on three general benchmarks and a medical
dataset for skin lesion classification reveals that our method consistently
performs better than representative baselines.
