  High-quality sentence embeddings are fundamental in many natural language
processing (NLP) tasks, such as semantic textual similarity (STS) and
retrieval-augmented generation (RAG). Nevertheless, most existing methods
leverage fixed-length embeddings from full-layer language models, which lack
the scalability to accommodate the diverse available resources across various
applications. Viewing this gap, we propose a novel sentence embedding model
$\mathrm{Espresso}$ $\mathrm{Sentence}$ $\mathrm{Embeddings}$ (ESE) with two
learning processes. First, the learn-to-express process encodes more salient
representations to lower layers. Second, the learn-to-compress process compacts
essential features into the initial dimensions using Principal Component
Analysis (PCA). This way, ESE can scale model depth via the former process and
embedding size via the latter. Extensive experiments on STS and RAG suggest
that ESE can effectively produce high-quality embeddings with less model depth
and embedding size, enhancing embedding inference efficiency.
