ACL Findings2026
SmartAD: Capacity-Aligned Agent Distillation for Small Language Models
Abstract
Large language models (LLMs) show strong reasoning and decision-making ability, but their high inference cost motivates transferring agentic skills to small language models (SLMs). Agent distillation trains SLMs on full reason-act-observe trajectories from a tool-using teacher, enabling SLMs to acquire the tool-use capabilities of large teacher models. However, some teacher-agent trajectories are simply hard for the student to learn, and their compatibility with the student can vary widely; moreover, a uniform token-level loss prevents SLMs from learning the tool-use patterns and final decisions that truly drive successful reasoning. Therefore, we propose SmartAD, a capacity-aligned agent distillation framework that improves both the distilled data and the supervision signal. SmartAD (i) selects, for each training example, the trajectory with the minimum negative log-likelihood among multiple correct teacher samples to obtain student-friendly training data, and (ii) applies a segment-weighted loss that emphasizes action execution and final decision spans over intermediate reasoning. Experiments on multihop QA and math benchmarks with 1.5B and 3B models show that SmartAD consistently outperforms all baselines. Overall, our method enables small models to learn the teacher's capabilities more easily and efficiently through trajectory selection and segment-weighted supervision, achieving capacity-aligned distillation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks cites LoRA: Low-Rank Adaptation of Large Language Models
- Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks cites Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
Builds on13
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model SizesCheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost et al.ACL Findings 2023 · 317 citations
- Measuring and Narrowing the Compositionality Gap in Language ModelsOfir Press, Muru Zhang, Sewon Min, Ludwig Schmidt et al.EMNLP Findings 2023 · 296 citations
- When More is Less: Understanding Chain-of-Thought Length in LLMsYuyang Wu, Yifei Wang, Ziyu Ye, Tianqi Du et al.ICLR 2026 · 225 citations
- Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive TasksMinki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi et al.NeurIPS 2023 · 128 citations
Related papers
- Distilling LLM Agent into Small Models with Retrieval and Code ToolsMinki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho et al.NeurIPS 2025 · 51 citations
- Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMsYuanjie Lyu, Chengyu Wang, Jun Huang, Tong XuICML 2026 · 9 citations
- Beyond Templates: Dynamic Adaptation of Reasoning Demonstrations via Feasibility-Aware ExplorationYong Wu, Weihang Pan, Ke Li, Chen Binhui et al.ACL Findings 2026
- Self-Evolutionary Reinforced Knowledge Distillation for Multi-Modal Tool-Use AgentsLei Shen, Chengyu Wang, Yuanjie Lyu, Yuanhao Yue et al.KDD 2026
- Long-Chain Reasoning Distillation via Adaptive Prefix AlignmentZhenghao Liu, Zhuoyang Wu, Xinze Li, Yukun Yan et al.ACL 2026 · 4 citations
