  Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent
achieving remarkable success in language understanding and generation. However,
such impressive capability typically comes with a substantial model size, which
presents significant challenges for autoregressive token-by-token generation.
To mitigate computation overload incurred during generation, several early-exit
and layer-dropping strategies have been proposed. Despite some promising
success due to the redundancy across LLMs layers on metrics like Rough-L/BLUE,
our careful knowledge-intensive evaluation unveils issues such as generation
collapse, hallucination of wrong facts, and noticeable performance drop even at
the trivial exit ratio of 10-15% of layers. We attribute these errors primarily
to ineffective handling of the KV cache through state copying during
early-exit. In this work, we observed the saturation of computationally
expensive feed-forward blocks of LLM layers and proposed FFN-SkipLLM, which is
a novel fine-grained skip strategy of autoregressive LLMs. More specifically,
FFN-SkipLLM is an input-adaptive feed-forward skipping strategy that can skip
25-30% of FFN blocks of LLMs with marginal change in performance on
knowledge-intensive generation tasks without any requirement to handle KV
cache. Our extensive experiments and ablation across benchmarks like MT-Bench,
Factoid-QA, and variable-length text summarization illustrate how our simple
and ease-at-use method can facilitate faster autoregressive decoding.
