  Current large language models (LLMs) primarily utilize next-token prediction
method for inference, which significantly impedes their processing speed. In
this paper, we introduce a novel inference methodology termed next-sentence
prediction, aimed at enhancing the inference efficiency of LLMs. We present
Sentence Variational Autoencoder (SentenceVAE), a tiny model consisting of a
Sentence Encoder and a Sentence Decoder. The Sentence Encoder can effectively
condense the information within a sentence into a singular token, while the
Sentence Decoder can reconstruct this compressed token back into sentence. By
integrating SentenceVAE into the input and output layers of LLMs, we develop
Sentence-level LLMs (SLLMs) that employ a sentence-by-sentence inference
method. In addition, the SentenceVAE module of SLLMS can maintain the integrity
of the original semantic content by segmenting the context into sentences,
thereby improving accuracy while boosting inference speed. Moreover, compared
to previous LLMs, SLLMs process fewer tokens over equivalent context length,
significantly reducing memory demands for self-attention computation and
facilitating the handling of longer context. Extensive experiments on Wanjuan
dataset have reveal that the proposed method can accelerate inference speed by
204~365%, reduce perplexity (PPL) to 46~75% of its original metric, and
decrease memory overhead by 86~91% for the equivalent context length, compared
to the token-by-token method.
