Lune

EMNLP Findings2023

Tuna: Instruction Tuning using Feedback from Large Language Models

Haoran Li, Yiran Liu, Xingxing Zhang, Wei Lu, Furu Wei

Abstract

Instruction tuning of open-source large language models (LLMs) like LLaMA, using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4, has proven to be a cost-effective way to align model behaviors with human preferences. However, the instruction-tuned model has only seen one response per instruction, lacking the knowledge of potentially better responses. In this paper, we propose finetuning an instruction-tuned LLM using our novel probabilistic ranking and contextual ranking approaches to increase the likelihood of generating better responses. Probabilistic ranking enables the instructiontuned model to inherit the relative rankings of high-quality and low-quality responses from the teacher LLM. On the other hand, learning with contextual ranking allows the model to refine its own response distribution using the contextual understanding ability of stronger LLMs. Furthermore, we apply probabilistic ranking and contextual ranking sequentially to the instruction-tuned LLM. The resulting model, which we call Tuna, consistently improves the performance on Super Natural Instructions (119 test tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results than several strong reinforcement learning baselines. Our code and data are available at https://github.com/microsoft/LMOps.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Language Models are Few-Shot Learners
  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Hyperion: Unveiling DApp Inconsistencies Using LLM and Dataflow-Guided Symbolic Execution cites Training language models to follow instructions with human feedback
  • MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models cites Training language models to follow instructions with human feedback
  • Finetuned Language Models are Zero-Shot Learners cites Language Models are Few-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Finetuned Language Models are Zero-Shot Learners
  • Training language models to follow instructions with human feedback cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
  • Forewarned is Forearmed: Harnessing LLMs for Data Synthesis via Failure-induced Exploration cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. MetaAligner: Towards Generalizable Multi-Objective Alignment of Language ModelsKailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang et al.NeurIPS 2024 · 49 citations
  2. Hyperion: Unveiling DApp Inconsistencies Using LLM and Dataflow-Guided Symbolic ExecutionShuo Yang, Xingwei Lin, Jiachi Chen, Qingyuan Zhong et al.ICSE 2025 · 6 citations
  3. Forewarned is Forearmed: Harnessing LLMs for Data Synthesis via Failure-induced ExplorationQintong Li, Jiahui Gao, Sheng Wang, Renjie Pi et al.ICLR 2025

Builds on17

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  4. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  5. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations

Related papers

  1. RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMsYue Yu, Wei Ping, Zihan Liu, Boxin Wang et al.NeurIPS 2024 · 321 citations
  2. Instruction Pre-Training: Language Models are Supervised Multitask LearnersDaixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi et al.EMNLP 2024 · 13 citations
  3. SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-ReflectionLiangxin Liu, Xuebo Liu, Derek F. Wong, Dongfang Li et al.NeurIPS 2024 · 49 citations
  4. LIMA: Less Is More for AlignmentChunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer et al.NeurIPS 2023 · 1,486 citations
  5. MAIN: Mutual Alignment Is Necessary for instruction tuningFanyi Yang, Jianfeng Liu, Xin Zhang, Haoyu Liu et al.EMNLP 2025
Dusk over the sea between two cliffs drawn in fine vertical lines