ACL Findings2025
Syntactic Control of Language Models by Posterior Inference
Vicky Xefteri, Tim Vieira, Ryan Cotterell, Afra Amini
Abstract
Controlling the syntactic structure of text generated by language models is valuable for applications requiring clarity, stylistic consistency, or interpretability, yet it remains a challenging task.In this paper, we argue that sampling algorithms based on the posterior inference can effectively enforce a target constituency structure during generation.Our approach combines sequential Monte Carlo, which estimates the posterior distribution by sampling from a proposal distribution, with a syntactic tagger that ensures that each generated token aligns with the desired syntactic structure.Our experiments with GPT2 and Llama3-8B models show that with an appropriate proposal distribution, we can improve syntactic accuracy, increasing the F1 score from 12.31 (GPT2-large) and 35.33 (Llama3-8B) to about 93 in both cases without compromising the language model's fluency.These results underscore both the complexity of syntactic control and the effectiveness of sampling algorithms, offering a promising approach for applications where precise control over syntax is essential.github.com/rycolab/syntactic-control
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on10
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- GeDi: Generative Discriminator Guided Sequence GenerationBen Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar et al.EMNLP Findings 2021 · 219 citations
- Synchromesh: Reliable Code Generation from Pre-trained Language ModelsGabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari et al.ICLR 2022 · 200 citations
- Constrained Language Models Yield Few-Shot Semantic ParsersRichard Shin, Christopher H. Lin, Sam Thomson, Charles Chen et al.EMNLP 2021 · 131 citations
- Probabilistic Inference in Language Models via Twisted Sequential Monte CarloStephen Zhao, Rob Brekelmans, Alireza Makhzani, Roger Baker GrosseICML 2024 · 61 citations
Related papers
- Syntactic and Semantic Control of Large Language Models via Sequential Monte CarloJoão Loula, Benjamin LeBrun, Li Du, Ben Lipkin et al.ICLR 2025
- Explicit Syntactic Guidance for Neural Text GenerationYafu Li, Leyang Cui, Jianhao Yan, Yongjing Yin et al.ACL 2023 · 4 citations
- NEUROSTRUCTURAL DECODING: Neural Text Generation with Structural ConstraintsMohaddeseh Bastan, Mihai Surdeanu, Niranjan BalasubramanianACL 2023
- Gradient-based Constrained Sampling from Language ModelsSachin Kumar, Biswajit Paria, Yulia TsvetkovEMNLP 2022 · 22 citations
- Guiding LLMs The Right Way: Fast, Non-Invasive Constrained GenerationLuca Beurer-Kellner, Marc Fischer, Martin T. VechevICML 2024 · 93 citations
