Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training

While large language models (LLMs) are equipped with longer text input capabilities than before, they struggle to seek correct information in long contexts. The "lost in the middle" problem challenges most LLMs, referring to the dramatic decline in accuracy when correct information is located in the middle. To overcome this crucial issue, this paper proposes to enhance the information searching and reflection ability of LLMs in long contexts via specially designed tasks called Position-Agnostic Multi-step QA (PAM QA). Trained with this task, our model excels in focusing more precisely on the desired information. Experimental results show substantial improvement in Multi-doc QA and other benchmarks, surpassing state-of-the-art models by a 13.7% absolute gain in shuffled settings and by 21.5% in the passage retrieval task. We release our model and code to promote related research in the community. 1