Lune

AAAI2026Top-tier venue

RESTL: Reinforcement Learning Guided by Multi-Aspect Rewards for Signal Temporal Logic Transformation

Yue Fang, Zhi Jin, Jie An, Hongshen Chen, Xiao-hong Chen, Naijun Zhan

Abstract

Signal Temporal Logic (STL) is a powerful formal language for specifying real-time specifications of Cyber-Physical Systems (CPS). Transforming specifications written in natural language into STL formulas automatically has attracted increasing attention. Existing rule-based methods depend heavily on rigid pattern matching and domain-specific knowledge, limiting their generalizability and scalability. Recently, Supervised Fine-Tuning (SFT) of large language models (LLMs) has been successfully applied to transform natural language into STL. However, the lack of fine-grained supervision on atomic proposition correctness, semantic fidelity, and formula readability often leads SFT-based methods to produce formulas misaligned with the intended meaning. To address these issues, we propose RESTL, a reinforcement learning (RL)-based framework for the transformation from natural language to STL. RESTL introduces multiple independently trained reward models that provide fine-grained, multi-faceted feedback from four perspectives, i.e., atomic proposition consistency, semantic alignment, formula succinctness, and symbol matching. These reward models are trained with a curriculum learning strategy to improve their feedback accuracy, and their outputs are aggregated into a unified signal that guides the optimization of the STL generator via Proximal Policy Optimization (PPO). Experimental results demonstrate that RESTL significantly outperforms state-of-the-art methods in both automatic metrics and human evaluations.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Enhancing Transformation from Natural Language to Signal Temporal Logic Using LLMs with Diverse External Knowledge cites NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models
  • NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models cites DeepSTL - From English Requirements to Signal Temporal Logic
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on6

  1. NL2TL: Transforming Natural Languages to Temporal Logics using Large Language ModelsYongchao Chen, Rujul Gandhi, Yang Zhang, Chuchu FanEMNLP 2023 · 48 citations
  2. DeepSTL - From English Requirements to Signal Temporal LogicJie He, Ezio Bartocci, Dejan Nickovic, Haris Isakovic et al.ICSE 2022 · 35 citations
  3. Reinforcement Learning Can Be More Efficient with Multiple RewardsChristoph Dann, Yishay Mansour, Mehryar MohriICML 2023 · 24 citations
  4. MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language ModelsYujing Wang, Hainan Zhang, Liang Pang, Binghui Guo et al.AAAI 2025 · 15 citations
  5. Enhancing Transformation from Natural Language to Signal Temporal Logic Using LLMs with Diverse External KnowledgeYue Fang, Zhi Jin, Jie An, Hongshen Chen et al.ACL Findings 2025 · 1 citation

Related papers

  1. Control Synthesis of Cyber-Physical Systems for Real-Time Specifications Through Causation-Guided Reinforcement LearningXiaochen Tang, Zhenya Zhang, Miaomiao Zhang, Jie AnRTSS 2025 · 1 citation
  2. Efficient SMT-Based Model Checking for Signal Temporal LogicJia Lee, Geunyeol Yu, Kyungmin BaeASE 2021 · 12 citations
  3. Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement LearningZijian Guo, Weichao Zhou, Wenchao LiICML 2024 · 7 citations
  4. ADARULE: LLM-Driven Natural Language to LTL Conversion via Pattern-Adaptive Rule InductionJiayi Hu, Jingling Sun, Chong Wang, Yihao Huang et al.ICSE 2026
  5. Supervised Reinforcement Learning: From Expert Trajectories to Step-wise ReasoningYihe Deng, I-Hung Hsu, Jun Yan, Zifeng Wang et al.ICLR 2026 · 11 citations
Dusk over the sea between two cliffs drawn in fine vertical lines