Lune

NeurIPS2025Top-tier venue

The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models

Ke Ji, Jiahao Xu, Tian Liang, Qiuzhi Liu, Zhiwei He, Xiaoyuan Liu, Xingyu Chen, Junying Chen, Benyou Wang, Zhaopeng Tu, Haitao Mi, Dong Yu

Abstract

Improving the reasoning capabilities of large language models (LLMs) typically requires supervised fine-tuning with labeled data or computationally expensive sampling. We introduce Unsupervised Prefix Fine-Tuning (UPFT), which leverages the observation of Prefix Self-Consistency -- the shared initial reasoning steps across diverse solution trajectories -- to enhance LLM reasoning efficiency. By training exclusively on the initial prefix substrings (as few as 8 tokens), UPFT removes the need for labeled data or exhaustive sampling. Experiments on reasoning benchmarks show that UPFT matches the performance of supervised methods such as Rejection Sampling Fine-Tuning, while reducing training time by 75% and sampling cost by 99%. Further analysis reveals that errors tend to appear in later stages of the reasoning process and that prefix-based training preserves the model's structural knowledge. This work demonstrates how minimal unsupervised fine-tuning can unlock substantial reasoning gains in LLMs, offering a scalable and resource-efficient alternative to conventional approaches.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

11 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Training language models to follow instructions with human feedback
  • Overthinking Reduction with Decoupled Rewards and Curriculum Data Scheduling cites Let's Verify Step by Step
  • SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment cites Training language models to follow instructions with human feedback
  • Let's Verify Step by Step cites STaR: Bootstrapping Reasoning With Reasoning
  • Let's Verify Step by Step cites Training language models to follow instructions with human feedback
  • Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilities cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers11

  1. Distilling LLM Agent into Small Models with Retrieval and Code ToolsMinki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho et al.NeurIPS 2025 · 51 citations
  2. SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early AlignmentWonje Jeung, Sangyeon Yoon, Minsuk Kahng, Albert NoNeurIPS 2025 · 31 citations
  3. Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational CapabilitiesWeixiang Zhao, Xingyu Sui, Jiahe Guo, Yulin Hu et al.AAAI 2026 · 21 citations
  4. Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in ReasoningWang Yang, Zirui Liu, Hongye Jin, Qingyu Yin et al.NeurIPS 2025 · 7 citations
  5. Overthinking Reduction with Decoupled Rewards and Curriculum Data SchedulingShuyang Jiang, Yusheng Liao, Ya Zhang, Yanfeng Wang et al.ICLR 2026 · 5 citations

Builds on10

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
  3. STaR: Bootstrapping Reasoning With ReasoningEric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. GoodmanNeurIPS 2022 · 1,126 citations
  4. The Flan Collection: Designing Data and Methods for Effective Instruction TuningShayne Longpre, Le Hou, Tu Vu, Albert Webson et al.ICML 2023 · 908 citations
  5. Large Language Models Cannot Self-Correct Reasoning YetJie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng et al.ICLR 2024 · 858 citations

Related papers

  1. Learning Diverse Responses with Prefix-Conditioned Supervised Fine-TuningZhiyuan Fan, Guanqiao Chen, Yanyi Huang, Mingkuan Zhao et al.ACL 2026
  2. Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy PrefixesAmrith Setlur, Zijian Wang, Andrew Cohen, Paria Rashidinejad et al.ICML 2026 · 13 citations
  3. UFT: Unifying Supervised and Reinforcement Fine-TuningMingyang Liu, Gabriele Farina, Asuman OzdaglarNeurIPS 2025 · 61 citations
  4. Self-Training Elicits Concise Reasoning in Large Language ModelsTergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang et al.ACL Findings 2025 · 3 citations
  5. Reasoning Quality Emerges Early: Data Curation for Reasoning ModelsHongyi Jin, Wenhan Yang, Meysam Ghaffari, Carlos Morato et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines