  The improvement of LLMs' instruction-following capabilities relies heavily on
the availability of high-quality instruction-response pairs. Unfortunately, the
current methods used to collect the pairs suffer from either unaffordable labor
costs or severe hallucinations in the self-generation of LLM. To tackle these
challenges, this paper proposes a scalable solution. It involves training LLMs
to generate instruction-response pairs based on human-written documents, rather
than relying solely on self-generation without context. Our proposed method not
only exploits the advantages of human-written documents in reducing
hallucinations but also utilizes an LLM to wrap the expression of documents,
which enables us to bridge the gap between various document styles and the
standard AI response. Experiments demonstrate that our method outperforms
existing typical methods on multiple benchmarks. In particular, compared to the
best-performing baseline, the LLM trained using our generated dataset exhibits
a 10\% relative improvement in performance on AlpacaEval, despite utilizing
only 1/5 of its training data. Furthermore, a comprehensive manual evaluation
validates the quality of the data we generated. Our trained wrapper is publicly
available at https://github.com/Bahuia/Dog-Instruct.
