Lune

ACL Findings2026

Scaling Unverifiable Rewards: A Case Study on Visual Insights

Shuyu Gan, James Mooney, Pan Hao, Renxiang Wang, Mingyi Hong, Qianwen Wang, Dongyeop Kang

Abstract

Large Language Model (LLM) agents can increasingly automate complex reasoning through Test-Time Scaling (TTS), an iterative refinement process guided by reward signals. However, many real-world tasks involve multistage pipelines whose final outcomes lack verifiable rewards or sufficient data to train robust reward models, making judge-based refinement prone to error accumulation across stages. We propose Selective TTS, a process-based refinement framework that scales inference across stages of a multi-agent pipeline, instead of repeatedly refining a single output over time as in prior work. By distributing compute across stages and pruning low-quality branches early using process-specific judgers, Selective TTS mitigates the judge drift and stabilizes refinement. Grounded in a data science workflow, we build an end-to-end multi-agent pipeline for generating visually insightful reports from a given dataset, and design a reliable LLM-based judge model that aligns with human experts (Kendall's τ =0.55) to evaluate them. Our proposed selective TTS then improves insight quality under a fixed compute budget, increasing mean scores from 61.64 (baseline) to 65.86 while reducing variance. We hope our findings serve as the first step toward scaling complex, open-ended tasks with unverifiable rewards like scientific discovery. Our code and generated reports are publicly available 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Self-Refine: Iterative Refinement with Self-Feedback
  • CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing cites Self-Refine: Iterative Refinement with Self-Feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on8

  1. Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
  2. Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
  3. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive CritiquingZhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen et al.ICLR 2024 · 699 citations
  4. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
  5. Fast Best-of-N Decoding via Speculative RejectionHanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang et al.NeurIPS 2024 · 144 citations

Related papers

  1. AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex TasksFali Wang, Hui Liu, Zhenwei Dai, Jingying Zeng et al.NeurIPS 2025 · 20 citations
  2. SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data ScienceWonduk Seo, Juhyeon Lee, Yanjun Shao, Qingshan Zhou et al.ACL 2026 · 8 citations
  3. AgentV-RL: Scaling Reward Modeling with Agentic VerifierJiazheng Zhang, Ziche Fu, Zhiheng Xi, Wenqing Jing et al.ACL Findings 2026
  4. IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement LearningHaohao Luo, Zexi Li, Yuexiang Xie, Wenhao Zhang et al.ICML 2026
  5. Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling EvaluatorsYilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong et al.ICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines