  Large Language Models (LLMs) excel in various natural language processing
tasks, but leveraging them for dense passage embedding remains challenging.
This is due to their causal attention mechanism and the misalignment between
their pre-training objectives and the text ranking tasks. Despite some recent
efforts to address these issues, existing frameworks for LLM-based text
embeddings have been limited by their support for only a limited range of LLM
architectures and fine-tuning strategies, limiting their practical application
and versatility. In this work, we introduce the Unified framework for Large
Language Model Embedding (ULLME), a flexible, plug-and-play implementation that
enables bidirectional attention across various LLMs and supports a range of
fine-tuning strategies. We also propose Generation-augmented Representation
Learning (GRL), a novel fine-tuning method to boost LLMs for text embedding
tasks. GRL enforces consistency between representation-based and
generation-based relevance scores, leveraging LLMs' powerful generative
abilities for learning passage embeddings. To showcase our framework's
flexibility and effectiveness, we release three pre-trained models from ULLME
with different backbone architectures, ranging from 1.5B to 8B parameters, all
of which demonstrate strong performance on the Massive Text Embedding
Benchmark. Our framework is publicly available at:
https://github.com/nlp-uoregon/ullme. A demo video for ULLME can also be found
at https://rb.gy/ws1ile.
