SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQA

<p>Question answering over hybrid contexts is a complex task, which requires the combina tion of information extracted from unstructured texts and structured tables in various ways. Re cently, In-Context Learning demonstrated sig nificant performance advances for reasoning tasks. In this paradigm, a large language model performs predictions based on a small set of supporting exemplars. The performance of In-Context Learning depends heavily on the selection procedure of the supporting exem plars, particularly in the case of HybridQA where considering the diversity of reasoning chains and the large size of the hybrid con texts becomes crucial. In this work, we present Selection of ExEmplars for hybrid Reasoning (SEER), a novel method for selecting a set of exemplars that is both representative and di verse. The key novelty of SEER is that it for mulates exemplar selection as a Knapsack Inte ger Linear Program. The Knapsack framework provides the flexibility to incorporate diversity constraints that prioritize exemplars with desir able attributes, and capacity constraints that en sure that the prompt size respects the provided capacity budgets. The effectiveness of SEER is demonstrated on FinQA and TAT-QA, two real-world benchmarks for HybridQA, where it outperforms previous exemplar selection meth ods.</p>