Lune

ACL Findings2026

Half-S: Halving the Scale for Near-Lossless 4-Bit LLM Training

Jinyang Du, Ruihao Gong, Linghan Ai, Zining Wang, Yunke Peng, Yao Wang, Lei Yan, Xuefei Wang, Yaoyuan Wang, Jinyang Guo, Dahua Lin, Xianglong Liu

Abstract

Training large language models (LLMs) at 4bit precision offers substantial efficiency gains but remains challenging due to the limited dynamic range and coarse numerical resolution. Existing 4-bit training pipelines typically rely on max-scaling, which is ill-suited for heavytailed LLM tensor distributions and leads to severe under-utilization of the FP4 quantization grid in the low-magnitude region. This effect causes pronounced representation collapse and large rounding errors for the values that dominate LLM computation. In this work, we derive the theoretically optimal scaling for FP4 under heavy-tailed inputs, revealing why max-scaling is intrinsically suboptimal. Guided by this analysis, we propose Half-S, a simple and efficient scaling strategy that uses half-scaling as a hardware-friendly default and falls back to an MSE-based clipping threshold when needed, yielding a close approximation to the theoretical optimum under real LLM statistics. Extensive experiments on large-scale pretraining and downstream finetuning show that Half-S consistently narrows the gap to BF16 in both convergence and final model quality, while preserving the efficiency benefits of 4-bit computation. Under native FP4 support, Half-S is estimated to provide up to 1.8× end-to-end training speedup. These results indicate that Half-S provides a simple and effective correction to max-scaling, substantially improving the stability and accuracy of 4-bit LLM training.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers cites Language Models are Few-Shot Learners
  • QLoRA: Efficient Finetuning of Quantized LLMs cites ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  • QLoRA: Efficient Finetuning of Quantized LLMs cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Solving Quantitative Reasoning Problems with Language Models cites Language Models are Few-Shot Learners
  • Solving Quantitative Reasoning Problems with Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  4. Solving Quantitative Reasoning Problems with Language ModelsAitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer et al.NeurIPS 2022 · 2,039 citations
  5. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale TransformersZhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu et al.NeurIPS 2022 · 816 citations

Related papers

  1. Quartet: Native FP4 Training Can Be Optimal for Large Language ModelsRoberto L. Castro, Andrei Panferov, Rush Tabesh, Oliver Sieberling et al.NeurIPS 2025 · 38 citations
  2. FP4 All the Way: Fully Quantized Training of Large Language ModelsBrian Chmiel, Maxim Fishman, Ron Banner, Daniel SoudryNeurIPS 2025 · 9 citations
  3. LLM-FP4: 4-Bit Floating-Point Quantized TransformersShih-Yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong et al.EMNLP 2023 · 34 citations
  4. Optimizing Large Language Model Training Using FP4 QuantizationRuizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao et al.ICML 2025
  5. TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier ControlYuxiang Chen, Yifan Liu, Xiaoming Xu, Pengle Zhang et al.ICML 2026 · 11 citations
Dusk over the sea between two cliffs drawn in fine vertical lines