Lune

EMNLP Findings2025

Learning to Instruct: Fine-Tuning a Task-Aware Instruction Optimizer for Black-Box LLMs

Yunzhe Qi, Jinjin Tian, Tianci Liu, Ruirui Li, Tianxin Wei, Hui Liu, Xianfeng Tang, Monica Xiao Cheng, Jingrui He

Abstract

The performance of Large Language Models (LLMs) critically depends on designing effective instructions, which is particularly challenging for black-box LLMs with inaccessible internal states. To this end, we introduce Learning to Instruct, a novel paradigm that formulates instruction optimization as an LLM fine-tuning objective for a white-box "instruction engineer" LLM, leveraging its rich learning capacity and vast pre-trained knowledge to enable efficient and effective instruction optimization. Within this paradigm, we propose Automatic Instruction Optimizer (AIO), a novel framework that fine-tunes a white-box LLM into a capable instruction engineer. AIO learns to optimize taskaware, human-comprehensible instructions by incorporating task nuances and feedback from the task-solving black-box LLM. To overcome the challenges of inaccessible black-box gradients and high API costs, AIO introduces a novel zeroth-order (ZO) gradient approximation mechanism guided by Thompson Sampling (TS), which reuses informative black-box LLM feedback for improved query efficiency. Extensive experiments show that AIO generally outperforms strong baselines in both effectiveness and efficiency, establishing Learning to Instruct as a promising new direction for blackbox LLM instruction optimization.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Language Models are Few-Shot Learners
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites LoRA: Low-Rank Adaptation of Large Language Models
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on29

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context LearningHaokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta et al.NeurIPS 2022 · 1,483 citations
  4. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  5. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-DistributionAnanya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma et al.ICLR 2022 · 911 citations

Related papers

  1. ACING: Actor-Critic for Instruction Learning in Black-Box LLMsSalma Kharrat, Fares Fourati, Marco CaniniEMNLP 2025 · 1 citation
  2. Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with TransformersXiaoqiang Lin, Zhaoxuan Wu, Zhongxiang Dai, Wenyang Hu et al.ICML 2024 · 26 citations
  3. InstructZero: Efficient Instruction Optimization for Black-Box Large Language ModelsLichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang et al.ICML 2024 · 64 citations
  4. PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMsJaewon Chu, Seunghun Lee, Hyunwoo J. KimNeurIPS 2025 · 5 citations
  5. Modality-Agnostic Zeroth-Order LoRA Fine-Tuning for Black-Box Prompt OptimizationXingchen Li, Jia Zhang, Tianxing Man, Wenkang Wang et al.KDD 2026
Dusk over the sea between two cliffs drawn in fine vertical lines