EMNLP Findings2025
Faster and Better LLMs via Latency-Aware Test-Time Scaling
Zili Wang, Tianyu Zhang, Haoli Bai, Lu Hou, Xianzhi Yu, Wulong Liu, Shiming Xiang, Lei Zhu
Abstract
Test-Time Scaling (TTS) has proven effective in improving the performance of Large Language Models (LLMs) during inference. However, existing research has overlooked the efficiency of TTS from a latency-sensitive perspective. Through a latency-aware evaluation of representative TTS methods, we demonstrate that a compute-optimal TTS does not always result in the lowest latency in scenarios where latency is critical. To address this gap and achieve latency-optimal TTS, we propose two key approaches by optimizing the concurrency configurations: (1) branch-wise parallelism, which leverages multiple concurrent inference branches, and (2) sequence-wise parallelism, enabled by speculative decoding. By integrating these two approaches and allocating computational resources properly to each, our latency-optimal TTS enables a 32B model to reach 82.3% accuracy on MATH-500 within 1 minute and a smaller 3B model to achieve 72.4% within 10 seconds. Our work emphasizes the importance of latency-aware TTS and
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites Fast Inference from Transformers via Speculative Decoding
- FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning cites EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
- FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning cites Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty cites Fast Inference from Transformers via Speculative Decoding
- Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Fast Inference from Transformers via Speculative Decoding
Cited by top-tier papers4
- Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference OptimizationXinyu Qiu, Heng Jia, Zhengwen Zeng, Shuheng Shen et al.CVPR 2026 · 4 citations
- Timely Machine: Awareness of Time Makes Test-Time Scaling AgenticYichuan Ma, Linyang Li, Yongkang Chen, Peiji Li et al.ACL 2026 · 3 citations
- TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding TasksHanwen Xu, Xuyao Huang, Yuzhe Liu, Zhijie DengACL 2026 · 2 citations
- FastTTS: Accelerating Test-Time Scaling for Edge LLM ReasoningHao Mark Chen, Zhiwen Mo, Guanxi Lu, Shuang Liang et al.ASPLOS 2026 · 1 citation
Builds on12
- Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
- Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng et al.ICML 2024 · 669 citations
- EAGLE: Speculative Sampling Requires Rethinking Feature UncertaintyYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangICML 2024 · 424 citations
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time TestYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangNeurIPS 2025 · 347 citations
Related papers
- ATTS: Asynchronous Test-Time Scaling via Conformal PredictionJing Xiong, Qiujiang Chen, Fanghua Ye, Zhongwei Wan et al.ICLR 2026 · 8 citations
- Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time ScalingShengyin Sun, Yiming Li, Xing Li, Yingzhao Lian et al.ICLR 2026 · 6 citations
- Attention-Level SpeculationJack Cai, Ammar Vora, Randolph Zhang, Mark O'Connor et al.ICML 2025
- TETRIS: Optimal Draft Token Selection for Batch Speculative DecodingZhaoxuan Wu, Zijian Zhou, Arun Verma, Alok Prakash et al.ACL 2025 · 7 citations
- Kinetics: Rethinking Test-Time Scaling LawRanajoy Sadhukhan, Zhuoming Chen, Haizhong Zheng, Beidi ChenNeurIPS 2025 · 2 citations
