Skill-Based Few-Shot Selection for In-Context Learning

In-context learning is the paradigm that adapts large language models to downstream tasks by providing a few examples. Few-shot selectionselecting appropriate examples for each test instance separately-is important for in-context learning. In this paper, we propose SKILL-KNN, a skill-based few-shot selection method for in-context learning. The key advantages of SKILL-KNN include: (1) it addresses the problem that existing methods based on pre-trained embeddings can be easily biased by surface natural language features that are not important for the target task; (2) it does not require training or fine-tuning of any models, making it suitable for frequently expanding or changing example banks. The key insight is to optimize the inputs fed into the embedding model, rather than tuning the model itself. Technically, SKILL-KNN generates the skill-based descriptions for each test case and candidate example by utilizing a pre-processing few-shot prompting, thus eliminating unimportant surface features. Experimental results across five cross-domain semantic parsing datasets and six backbone models show that SKILL-KNN significantly outperforms existing methods. * Work done during the internship at Microsoft. DB Schema: employee (name, age, city, …) … Question: Which cities do more than one employee under age 30 come from? Prompting-Based Rewriting Off-the-Shelf Embedding Model Skill-Based Descriptions Input Query Candidate 1:This task requires the greater-than and less-than constraints. Candidate 2:This task requires to apply two constraints on one selected column. Candidate … … Example Bank Skill-Based Selection DB Schema: cinema (name, capacity, location, …) … Question: Find the locations that have more than one movie theater with capacity above 300. SQL Query: SELECT … WHERE capacity > 300 GROUP BY location HAVING count() > 1 DB Schema: endowment (school, donator, amount, …) … Question : Find the number of schools that have more than one donator whose donation amount is less than 8.5. SQL Query: SELECT ... WHERE amount > 8.5 GROUP BY school HAVING count() > 1 … Context DB Schema: employee (name, age, city, …) … Question: Which cities do more than one employee under age 30 come from? In-Context Learning SQL Query: SELECT city FROM employee WHERE age < 30 GROUP BY city HAVING count(*) > 1