  Finetuning large language models with a variety of instruction-response pairs
has enhanced their capability to understand and follow instructions. Current
instruction tuning primarily relies on teacher models or human intervention to
generate and refine the instructions and responses, which are costly,
non-sustainable, and may lack diversity. In this paper, we introduce Mosaic
Instruction Tuning (Mosaic-IT), a human/model-free method that can efficiently
create rich and diverse augmentations from existing instruction tuning data to
enhance the finetuned LLM.Mosaic-IT randomly concatenates multiple instruction
data into one and trains the model to produce the corresponding responses with
predefined higher-level meta-instructions to strengthen its multi-step
instruction-following and format-following skills. Our extensive evaluations
demonstrate a superior performance and training efficiency of Mosaic-IT, which
achieves consistent performance improvements over various benchmarks and an 80%
reduction in training costs compared with original instruction tuning. Our
codes and data are available at https://github.com/tianyi-lab/Mosaic-IT.
