  Instruction tuning has shown its ability to not only enhance zero-shot
generalization across various tasks but also its effectiveness in improving the
performance of specific tasks. A crucial aspect in instruction tuning for a
particular task is a strategic selection of related tasks that offer meaningful
supervision, thereby enhancing efficiency and preventing performance
degradation from irrelevant tasks. Our research reveals that leveraging
instruction information \textit{alone} enables the identification of pertinent
tasks for instruction tuning. This approach is notably simpler compared to
traditional methods that necessitate complex measurements of pairwise
transferability between tasks or the creation of data samples for the target
task. Furthermore, by additionally learning the unique instructional template
style of the meta-dataset, we observe an improvement in task selection
accuracy, which contributes to enhanced overall performance. Experimental
results demonstrate that training on a small set of tasks, chosen solely based
on the instructions, leads to substantial performance improvements on
benchmarks like P3, Big-Bench, NIV2, and Big-Bench Hard. Significantly, these
improvements exceed those achieved by prior task selection methods,
highlighting the efficacy of our approach.
