Lune

EMNLP Findings2025

Faster and Better LLMs via Latency-Aware Test-Time Scaling

Zili Wang, Tianyu Zhang, Haoli Bai, Lu Hou, Xianzhi Yu, Wulong Liu, Shiming Xiang, Lei Zhu

Abstract

Test-Time Scaling (TTS) has proven effective in improving the performance of Large Language Models (LLMs) during inference. However, existing research has overlooked the efficiency of TTS from a latency-sensitive perspective. Through a latency-aware evaluation of representative TTS methods, we demonstrate that a compute-optimal TTS does not always result in the lowest latency in scenarios where latency is critical. To address this gap and achieve latency-optimal TTS, we propose two key approaches by optimizing the concurrency configurations: (1) branch-wise parallelism, which leverages multiple concurrent inference branches, and (2) sequence-wise parallelism, enabled by speculative decoding. By integrating these two approaches and allocating computational resources properly to each, our latency-optimal TTS enables a 32B model to reach 82.3% accuracy on MATH-500 within 1 minute and a smaller 3B model to achieve 72.4% within 10 seconds. Our work emphasizes the importance of latency-aware TTS and

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
  • EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
  • EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites Fast Inference from Transformers via Speculative Decoding
  • FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning cites EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
  • FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning cites Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
  • EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty cites Fast Inference from Transformers via Speculative Decoding
  • Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Fast Inference from Transformers via Speculative Decoding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference OptimizationXinyu Qiu, Heng Jia, Zhengwen Zeng, Shuheng Shen et al.CVPR 2026 · 4 citations
  2. Timely Machine: Awareness of Time Makes Test-Time Scaling AgenticYichuan Ma, Linyang Li, Yongkang Chen, Peiji Li et al.ACL 2026 · 3 citations
  3. TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding TasksHanwen Xu, Xuyao Huang, Yuzhe Liu, Zhijie DengACL 2026 · 2 citations
  4. FastTTS: Accelerating Test-Time Scaling for Edge LLM ReasoningHao Mark Chen, Zhiwen Mo, Guanxi Lu, Shuang Liang et al.ASPLOS 2026 · 1 citation

Builds on12

  1. Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
  2. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
  3. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng et al.ICML 2024 · 669 citations
  4. EAGLE: Speculative Sampling Requires Rethinking Feature UncertaintyYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangICML 2024 · 424 citations
  5. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time TestYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangNeurIPS 2025 · 347 citations

Related papers

  1. ATTS: Asynchronous Test-Time Scaling via Conformal PredictionJing Xiong, Qiujiang Chen, Fanghua Ye, Zhongwei Wan et al.ICLR 2026 · 8 citations
  2. Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time ScalingShengyin Sun, Yiming Li, Xing Li, Yingzhao Lian et al.ICLR 2026 · 6 citations
  3. Attention-Level SpeculationJack Cai, Ammar Vora, Randolph Zhang, Mark O'Connor et al.ICML 2025
  4. TETRIS: Optimal Draft Token Selection for Batch Speculative DecodingZhaoxuan Wu, Zijian Zhou, Arun Verma, Alok Prakash et al.ACL 2025 · 7 citations
  5. Kinetics: Rethinking Test-Time Scaling LawRanajoy Sadhukhan, Zhuoming Chen, Haizhong Zheng, Beidi ChenNeurIPS 2025 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines