Lune

EMNLP Findings2023

Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing

Shufan Wang, Sébastien Jean, Sailik Sengupta, James Gung, Nikolaos Pappas, Yi Zhang

Abstract

In executable task-oriented semantic parsing, the system aims to translate users' utterances in natural language to machine-interpretable programs (API calls) that can be executed according to pre-defined API specifications. With the popularity of Large Language Models (LLMs), in-context learning offers a strong baseline for such scenarios, especially in data-limited regimes (Hu et al., 2022;Shin et al., 2021). However, LLMs are known to hallucinate and therefore pose a formidable challenge in constraining generated content (Parikh et al., 2020). Thus, it remains uncertain if LLMs can effectively perform task-oriented utterance-to-API generation where respecting API's structural and task-specific constraints is crucial. In this work, we seek to measure, analyze and mitigate such constraints violations. First, we identify the categories of various constraints in obtaining API-semantics from task-oriented utterances, and define fine-grained metrics that complement traditional ones. Second, we leverage these metrics to conduct a detailed error analysis of constraints violations seen in state-of-theart LLMs, which motivates us to investigate two popular mitigation strategies-Semantic-Retrieval of Demonstrations (SRD) and APIaware Constrained Decoding (API-CD). Our experiments show that these strategies are effective at reducing constraints violations and improving the quality of the generated API calls, but require careful consideration given their implementation complexity and latency.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context Learning cites Language Models are Few-Shot Learners
  • Constrained Language Models Yield Few-Shot Semantic Parsers cites Language Models are Few-Shot Learners
  • ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs cites Toolformer: Language Models Can Teach Themselves to Use Tools
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context LearningYubo Mai, Zhipeng Gao, Xing Hu, Lingfeng Bao et al.FSE 2024 · 4 citations

Builds on13

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
  3. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIsYujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu et al.ICLR 2024 · 1,469 citations
  4. Constrained Language Models Yield Few-Shot Semantic ParsersRichard Shin, Christopher H. Lin, Sam Thomson, Charles Chen et al.EMNLP 2021 · 131 citations
  5. Grounded Adaptation for Zero-shot Executable Semantic ParsingVictor Zhong, Mike Lewis, Sida I. Wang, Luke ZettlemoyerEMNLP 2020 · 85 citations

Related papers

  1. Learning API Functionality from In-Context Demonstrations for Tool-based AgentsBhrij Patel, Ashish Jagmohan, Aditya VempatyEMNLP Findings 2025 · 1 citation
  2. In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination MitigationShiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu et al.ICML 2024 · 49 citations
  3. FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and RerankingZhuoer Wang, Leonardo F. R. Ribeiro, Alexandros Papangelis, Rohan Mukherjee et al.EMNLP Findings 2024
  4. The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained DecodingAvinash Reddy, Thayne Walker, Jaime Ide, Amrit Singh BediICML 2026 · 6 citations
  5. Constrained Decoding of Diffusion LLMs with Context-Free GrammarsNiels Mündler, Jasper Dekoninck, Martin VechevICLR 2026 · 18 citations
Dusk over the sea between two cliffs drawn in fine vertical lines