Making Text Embedders Few-Shot Learners

Large language models (LLMs) with decoder-only architectures have demonstrated exceptional text-generation capabilities across a variety of tasks. Some researchers have also adapted these models for text representation tasks. However, in text representation tasks, these models often face performance degradation on unseen tasks. In-context learning (ICL), which leverages examples provided in the input context, enables LLMs to handle unseen tasks effectively. Inspired by this, we aim to fully utilize the inherent properties of LLMs to enhance text representation performance across different tasks through the ICL approach. In this paper, we introduce a simple yet effective training strategy, which significantly improves text representation capabilities. Unlike previous models that prepend task instructions to the text, our method randomly samples a varying number of examples during training, endowing the embedding model with in-context learning abilities while maintaining its zero-shot capabilities. This approach does not require additional data construction or modifications to the model architecture. On the contrary, we find that some popular modifications to the model, such as bidirectional attention, can degrade performance, undermining the inherent characteristics of LLMs. We have publicly released our method at this repo. * Co-first authors † Corresponding authors, with Zheng Liu as the project lead Similarity Score Once upon a time, in a blooming meadow, a group of rabbits were happily racing each other. Their playful chase led them to a hidden, glowing burrow. Inside, they discovered an enchanted world where animals spoke and wishes came true, a secret haven of endless adventures. On a meadow, a group of rabbits are running, with an eagle chasing them from behind. To survive, the rabbits must run as fast as they can. Candidates Query A group of rabbits are running. Scene: A cat is chasing a mouse through a castle. Fairy Tale: In an ancient castle, a mouse named Max and a cat named Sir Whiskers stumbled upon a secret chamber with a magical crystal. Instead of continuing their chase, they called a truce to protect the crystal. Together, they used its magic to bring prosperity and harmony to the castle. Scene: A frog is sitting on a lilypad under a moonlit sky. Fairy Tale: Under a moonlit sky, a cursed prince in the form of a frog sat on a lilypad. A kind maiden named Lila came by and, moved by his sorrow, kissed him. The curse was broken, and the frog transformed into a prince. They married and ruled a kingdom happily ever after. Scene: A young girl discovers an old, dusty book in an attic. Fairy Tale: Once upon a time, a curious young girl named Eliza found an old, dusty book in her grandmother's attic. As she opened it, she was transported into a magical realm where she had to help a brave knight save a cursed kingdom. Together, they broke the curse and restored peace. Given a scene, retrieve the fairy tale that unfolds with this scene.