Lune

EMNLP Findings2024

Reformatted Alignment

Run-Ze Fan, Xuefeng Li, Haoyang Zou, Junlong Li, Shwai He, Ethan Chern, Jiewen Hu, Pengfei Liu

Abstract

The quality of finetuning data is crucial for aligning large language models (LLMs) with human values. Current methods to improve data quality are either labor-intensive or prone to factual errors caused by LLM hallucinations. This paper explores elevating the quality of existing instruction data to better align with human values, introducing a simple and effective approach named REALIGN, which reformats the responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence. This approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques. Experimentally, REALIGN significantly boosts the general alignment ability, math reasoning, factuality, and readability of the LLMs.

Encouragingly, without introducing any additional data or advanced training techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy. Additionally, a mere 5% of REALIGN data yields a 67% boost in general alignment ability measured by the Alpaca dataset. This work highlights the need for further research into the science and mechanistic interpretability of LLMs. We have made the associated code, defined criteria, and data publicly accessible to support future studies at https://github.com/GAIR-NLP/ReAlign.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents cites Training language models to follow instructions with human feedback
  • What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning cites Training language models to follow instructions with human feedback
  • Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models cites Language Models are Few-Shot Learners
  • Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models cites Training language models to follow instructions with human feedback
  • Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models cites Cross-Task Generalization via Natural Language Crowdsourcing Instructions
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
  • Training language models to follow instructions with human feedback cites Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Tool Learning in the Wild: Empowering Language Models as Automatic Tool AgentsZhengliang Shi, Shen Gao, Lingyong Yan, Yue Feng et al.WWW 2025 · 59 citations
  2. Are Large Language Models Good at Utility Judgments?Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke et al.SIGIR 2024 · 20 citations

Builds on11

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Cross-Task Generalization via Natural Language Crowdsourcing InstructionsSwaroop Mishra, Daniel Khashabi, Chitta Baral, Hannaneh HajishirziACL 2022 · 887 citations
  4. Self-Play Fine-Tuning Converts Weak Language Models to Strong Language ModelsZixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji et al.ICML 2024 · 527 citations
  5. What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction TuningWei Liu, Weihao Zeng, Keqing He, Yong Jiang et al.ICLR 2024 · 369 citations

Related papers

  1. Question Translation Training for Better Multilingual ReasoningWenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She et al.ACL Findings 2024 · 7 citations
  2. Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-TuningHao Zhao, Maksym Andriushchenko, Francesco Croce, Nicolas FlammarionICML 2024 · 96 citations
  3. FLAME : Factuality-Aware Alignment for Large Language ModelsSheng-Chieh Lin, Luyu Gao, Barlas Oguz, Wenhan Xiong et al.NeurIPS 2024 · 63 citations
  4. From Selection to Refinement: Iterative Optimization for Instruction DataHang Hu, Ziyan Liu, Rujie Wen, Ruihui Hou et al.ACL 2026
  5. MARIO: MAth Reasoning with code Interpreter Output - A Reproducible PipelineMinpeng Liao, Chengxi Li, Wei Luo, Jing Wu et al.ACL Findings 2024 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines