  We introduce a decoder-decoder architecture, YOCO, for large language models,
which only caches key-value pairs once. It consists of two components, i.e., a
cross-decoder stacked upon a self-decoder. The self-decoder efficiently encodes
global key-value (KV) caches that are reused by the cross-decoder via
cross-attention. The overall model behaves like a decoder-only Transformer,
although YOCO only caches once. The design substantially reduces GPU memory
demands, yet retains global attention capability. Additionally, the computation
flow enables prefilling to early exit without changing the final output,
thereby significantly speeding up the prefill stage. Experimental results
demonstrate that YOCO achieves favorable performance compared to Transformer in
various settings of scaling up model size and number of training tokens. We
also extend YOCO to 1M context length with near-perfect needle retrieval
accuracy. The profiling results show that YOCO improves inference memory,
prefill latency, and throughput by orders of magnitude across context lengths
and model sizes. Code is available at https://aka.ms/YOCO.
