  Deep neural models have hitherto achieved significant performances on
numerous classification tasks, but meanwhile require sufficient manually
annotated data. Since it is extremely time-consuming and expensive to annotate
adequate data for each classification task, learning an empirically effective
model with generalization on small dataset has received increased attention.
Existing efforts mainly focus on transferring task-relevant knowledge from
other similar data to tackle the issue. These approaches have yielded
remarkable improvements, yet neglecting the fact that the task-irrelevant
features could bring out massive negative transfer effects. To date, no
large-scale studies have been performed to investigate the impact of
task-irrelevant features, let alone the utilization of this kind of features.
In this paper, we firstly propose Task-Irrelevant Transfer Learning (TIRTL) to
exploit task-irrelevant features, which mainly are extracted from
task-irrelevant labels. Particularly, we suppress the expression of
task-irrelevant information and facilitate the learning process of
classification. We also provide a theoretical explanation of our method. In
addition, TIRTL does not conflict with those that have previously exploited
task-relevant knowledge and can be well combined to enable the simultaneous
utilization of task-relevant and task-irrelevant features for the first time.
In order to verify the effectiveness of our theory and method, we conduct
extensive experiments on facial expression recognition and digit recognition
tasks. Our source code will be also available in the future for
reproducibility.
