RESTL: Reinforcement Learning Guided by Multi-Aspect Rewards for Signal Temporal Logic Transformation
Yue Fang, Zhi Jin, Jie An, Hongshen Chen, Xiao-hong Chen, Naijun Zhan
Abstract
Signal Temporal Logic (STL) is a powerful formal language for specifying real-time specifications of Cyber-Physical Systems (CPS). Transforming specifications written in natural language into STL formulas automatically has attracted increasing attention. Existing rule-based methods depend heavily on rigid pattern matching and domain-specific knowledge, limiting their generalizability and scalability. Recently, Supervised Fine-Tuning (SFT) of large language models (LLMs) has been successfully applied to transform natural language into STL. However, the lack of fine-grained supervision on atomic proposition correctness, semantic fidelity, and formula readability often leads SFT-based methods to produce formulas misaligned with the intended meaning. To address these issues, we propose RESTL, a reinforcement learning (RL)-based framework for the transformation from natural language to STL. RESTL introduces multiple independently trained reward models that provide fine-grained, multi-faceted feedback from four perspectives, i.e., atomic proposition consistency, semantic alignment, formula succinctness, and symbol matching. These reward models are trained with a curriculum learning strategy to improve their feedback accuracy, and their outputs are aggregated into a unified signal that guides the optimization of the STL generator via Proximal Policy Optimization (PPO). Experimental results demonstrate that RESTL significantly outperforms state-of-the-art methods in both automatic metrics and human evaluations.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Enhancing Transformation from Natural Language to Signal Temporal Logic Using LLMs with Diverse External Knowledge cites NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models
- NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models cites DeepSTL - From English Requirements to Signal Temporal Logic
Builds on6
- NL2TL: Transforming Natural Languages to Temporal Logics using Large Language ModelsYongchao Chen, Rujul Gandhi, Yang Zhang, Chuchu FanEMNLP 2023 · 48 citations
- DeepSTL - From English Requirements to Signal Temporal LogicJie He, Ezio Bartocci, Dejan Nickovic, Haris Isakovic et al.ICSE 2022 · 35 citations
- Reinforcement Learning Can Be More Efficient with Multiple RewardsChristoph Dann, Yishay Mansour, Mehryar MohriICML 2023 · 24 citations
- MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language ModelsYujing Wang, Hainan Zhang, Liang Pang, Binghui Guo et al.AAAI 2025 · 15 citations
- Enhancing Transformation from Natural Language to Signal Temporal Logic Using LLMs with Diverse External KnowledgeYue Fang, Zhi Jin, Jie An, Hongshen Chen et al.ACL Findings 2025 · 1 citation
Related papers
- Control Synthesis of Cyber-Physical Systems for Real-Time Specifications Through Causation-Guided Reinforcement LearningXiaochen Tang, Zhenya Zhang, Miaomiao Zhang, Jie AnRTSS 2025 · 1 citation
- Efficient SMT-Based Model Checking for Signal Temporal LogicJia Lee, Geunyeol Yu, Kyungmin BaeASE 2021 · 12 citations
- Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement LearningZijian Guo, Weichao Zhou, Wenchao LiICML 2024 · 7 citations
- ADARULE: LLM-Driven Natural Language to LTL Conversion via Pattern-Adaptive Rule InductionJiayi Hu, Jingling Sun, Chong Wang, Yihao Huang et al.ICSE 2026
- Supervised Reinforcement Learning: From Expert Trajectories to Step-wise ReasoningYihe Deng, I-Hung Hsu, Jun Yan, Zifeng Wang et al.ICLR 2026 · 11 citations
