Lune

ACL Findings2025

Syntactic Control of Language Models by Posterior Inference

Vicky Xefteri, Tim Vieira, Ryan Cotterell, Afra Amini

Abstract

Controlling the syntactic structure of text generated by language models is valuable for applications requiring clarity, stylistic consistency, or interpretability, yet it remains a challenging task.In this paper, we argue that sampling algorithms based on the posterior inference can effectively enforce a target constituency structure during generation.Our approach combines sequential Monte Carlo, which estimates the posterior distribution by sampling from a proposal distribution, with a syntactic tagger that ensures that each generated token aligns with the desired syntactic structure.Our experiments with GPT2 and Llama3-8B models show that with an appropriate proposal distribution, we can improve syntactic accuracy, increasing the F1 score from 12.31 (GPT2-large) and 35.33 (Llama3-8B) to about 93 in both cases without compromising the language model's fluency.These results underscore both the complexity of syntactic control and the effectiveness of sampling algorithms, offering a promising approach for applications where precise control over syntax is essential.github.com/rycolab/syntactic-control

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Synchromesh: Reliable Code Generation from Pre-trained Language Models cites Constrained Language Models Yield Few-Shot Semantic Parsers
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Transducing Language ModelsVésteinn Snæbjarnarson, Samuel Kiegeland, Tianyu Liu, Reda Boumasmoud et al.ICLR 2026 · 3 citations

Builds on10

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. GeDi: Generative Discriminator Guided Sequence GenerationBen Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar et al.EMNLP Findings 2021 · 219 citations
  3. Synchromesh: Reliable Code Generation from Pre-trained Language ModelsGabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari et al.ICLR 2022 · 200 citations
  4. Constrained Language Models Yield Few-Shot Semantic ParsersRichard Shin, Christopher H. Lin, Sam Thomson, Charles Chen et al.EMNLP 2021 · 131 citations
  5. Probabilistic Inference in Language Models via Twisted Sequential Monte CarloStephen Zhao, Rob Brekelmans, Alireza Makhzani, Roger Baker GrosseICML 2024 · 61 citations

Related papers

  1. Syntactic and Semantic Control of Large Language Models via Sequential Monte CarloJoão Loula, Benjamin LeBrun, Li Du, Ben Lipkin et al.ICLR 2025
  2. Explicit Syntactic Guidance for Neural Text GenerationYafu Li, Leyang Cui, Jianhao Yan, Yongjing Yin et al.ACL 2023 · 4 citations
  3. NEUROSTRUCTURAL DECODING: Neural Text Generation with Structural ConstraintsMohaddeseh Bastan, Mihai Surdeanu, Niranjan BalasubramanianACL 2023
  4. Gradient-based Constrained Sampling from Language ModelsSachin Kumar, Biswajit Paria, Yulia TsvetkovEMNLP 2022 · 22 citations
  5. Guiding LLMs The Right Way: Fast, Non-Invasive Constrained GenerationLuca Beurer-Kellner, Marc Fischer, Martin T. VechevICML 2024 · 93 citations
Dusk over the sea between two cliffs drawn in fine vertical lines