Lune

ACL Findings2025

Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teaching

Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Yipeng Zhang, Haitao Mi, Helen M. Meng

Abstract

Large language models (LLMs) often struggle to provide up-to-date information due to their one-time training and the constantly evolving nature of the world. To keep LLMs current, existing approaches typically involve continued pre-training on new documents. However, they frequently face difficulties in extracting stored knowledge. Motivated by the remarkable success of the Feynman Technique in efficient human learning, we introduce SELF-TUNING, a learning framework aimed at improving an LLM's ability to effectively acquire new knowledge from unseen raw documents through self-teaching. Specifically, we develop a SELF-TEACHING strategy that augments the documents with a set of knowledge-intensive tasks created in a self-supervised manner, focusing on three crucial aspects: memorization, comprehension, and self-reflection. Additionally, we introduce three Wiki-Newpages-2023-QA datasets to facilitate an in-depth analysis of an LLM's knowledge acquisition ability concerning memorization, extraction, and reasoning. Extensive experimental results on various models, e.g., LLAMA2-7B reveal that SELF-TUNING consistently exhibits superior performance across all knowledge acquisition tasks and excels in preserving previous knowledge. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • NILE: Internal Consistency Alignment in Large Language Models cites Finetuned Language Models are Zero-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. NILE: Internal Consistency Alignment in Large Language ModelsMinda Hu, Qiyuan Zhang, Yufei Wang, Bowei He et al.EMNLP 2025

Builds on16

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  4. Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
  5. Fast Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn et al.ICLR 2022 · 527 citations

Related papers

  1. Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMsOded Ovadia, Menachem Brief, Moshik Mishaeli, Oren ElishaEMNLP 2024 · 89 citations
  2. Enhancing LLM Knowledge Learning through GeneralizationMingkang Zhu, Xi Chen, Zhongdao Wang, Bei Yu et al.EMNLP Findings 2025 · 2 citations
  3. Unlocking LLMs' Self-Improvement Capacity with Autonomous Learning for Domain AdaptationKe Ji, Junying Chen, Anningzhe Gao, Wenya Xie et al.ACL Findings 2025
  4. Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter LevelsJunjie Ye, Yuming Yang, Yang Nan, Shuo Li et al.EMNLP 2025
  5. Data Doping or True Intelligence? Evaluating the Transferability of Injected Knowledge in LLMsEssa Jan, Moiz Ali, Muhammad Saram Hassan, Muhammad Fareed Zaffar et al.EMNLP Findings 2025
Dusk over the sea between two cliffs drawn in fine vertical lines