Lune

ACL Findings2025

Self-Foveate: Enhancing Diversity and Difficulty of Synthesized Instructions from Unsupervised Text via Multi-Level Foveation

Mingzhe Li, Xin Lu, Yanyan Zhao

Abstract

Large language models (LLMs) with instruction following capabilities have demonstrated impressive problem-solving abilities. While synthesizing instructional data from unsupervised text has become a common approach for training such models, conventional methods rely heavily on human effort for data annotation. Although existing automated synthesis paradigms have alleviated this constraint, they still exhibit significant limitations in ensuring adequate diversity and difficulty of synthesized instructions. To address these challenges, we propose SELF-FOVEATE, an innovative LLMdriven method for instruction synthesis. This approach introduces a "Micro-Scatter-Macro" multi-level foveation methodology that effectively guides the LLM to deeply excavate finegrained information embedded in unsupervised text, thereby enhancing both the diversity and difficulty of synthesized instructions. Comprehensive experiments across multiple unsupervised corpora and diverse model architectures validate the effectiveness and superiority of our proposed method. We publicly release our data and codes: https://github.com/Mubuky/Self- Foveate

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • LIMA: Less Is More for Alignment cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • LIMA: Less Is More for Alignment cites Finetuned Language Models are Zero-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on23

  1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
  2. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  3. FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
  4. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
  5. LIMA: Less Is More for AlignmentChunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer et al.NeurIPS 2023 · 1,486 citations

Related papers

  1. FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs OnlyHe Zhu, Yifan Ding, Yicheng Tao, Zhiwen Ruan et al.ACL Findings 2025 · 2 citations
  2. DecIF: Improving Instruction-Following through DecompositionTingfeng Hui, Pengyu Zhu, Bowen Ping, Ling Tang et al.ACL 2026
  3. Tree-of-Evolution: Tree-Structured Instruction Evolution for Code Generation in Large Language ModelsZiyang Luo, Kaixin Li, Hongzhan Lin, Yuchen Tian et al.ACL 2025 · 4 citations
  4. Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language ModelsGuanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia et al.ICLR 2025 · 4 citations
  5. REInstruct: Building Instruction Data from Unlabeled CorpusShu Chen, Xinyan Guan, Yaojie Lu, Hongyu Lin et al.ACL Findings 2024 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines