  The burgeoning size of Large Language Models (LLMs) has led to enhanced
capabilities in generating responses, albeit at the expense of increased
inference times and elevated resource demands. Existing methods of
acceleration, predominantly hinged on knowledge distillation, generally
necessitate fine-tuning of considerably large models, such as Llama-7B, posing
a challenge for average users. Furthermore, present techniques for expediting
inference and reducing costs operate independently. To address these issues, we
introduce a novel and intuitive Guidance-based Knowledge Transfer (GKT)
framework. This approach leverages a larger LLM as a ''teacher'' to create
guidance prompts, paired with a smaller ''student'' model to finalize
responses. Remarkably, GKT requires no fine-tuning and doesn't necessitate the
teacher and student models to have the same vocabulary, allowing for extensive
batch generation to accelerate the process while ensuring user customization.
GKT can be seamlessly integrated into cloud-edge collaboration architectures,
and is versatile enough for plug-and-play application across various models. It
excels in both efficiency and affordability, epitomizing a ''cheap and
cheerful'' solution. GKT achieves a maximum accuracy improvement of 14.18%,
along with a 10.72 times speed-up on GSM8K and an accuracy improvement of 14.00
% along with a 7.73 times speed-up in CSQA. When utilizing ChatGPT as teacher
model and Llama2-70B as the student model, we can achieve 95.00% of ChatGPT's
performance at 52% of the cost. The results highlight substantial enhancements
in accuracy and processing speed on the GSM8K and CSQA datasets, surpassing the
performance of using either the student or teacher models in isolation.
