Lune

EMNLP Findings2025

FedCoT: Federated Chain-of-Thought Distillation for Large Language Models

Tao Fan, Weijing Chen, Yan Kang, Guoqiang Ma, Hanlin Gu, Yuanfeng Song, Lixin Fan, Qiang Yang

Abstract

Large Language Models (LLMs) have emerged as a transformative force in artificial intelligence, demonstrating exceptional proficiency across various tasks.However, their deployment in resource-constrained environments and concerns over user data privacy pose significant challenges.In contrast, Small Language Models (SLMs) offer computational efficiency but often lag in performance.To address these issues, we propose FedCoT, a federated framework designed for the Chainof-Thought (CoT) distillation of knowledge from LLMs to SLMs, while ensuring the preservation of clients' data privacy.FedCoT ensures secure and efficient knowledge transfer from an LLM on a high-powered server to an SLM on a resource-constrained client, while adhering to privacy requirements.Leveraging perturbed prompts and rationales generated through the CoT approach, the framework enhances the performance of the client's SLM without compromising user data privacy within a multi-task learning framework.We propose two privacy protection strategies: the Exponential Mechanism Strategy and the Adaptive Exponential Mechanism Strategy, which balance user prompt privacy and the usability of rationales.Empirical evaluation on various text generation tasks demonstrates the effectiveness of FedCoT in training taskspecific SLMs with enhanced performance while prioritizing data privacy protection.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes cites Large Language Models are Zero-Shot Reasoners
  • Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • A Survey on In-context Learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  3. A Survey on In-context LearningQingxiu Dong, Lei Li, Damai Dai, Ce Zheng et al.EMNLP 2024 · 479 citations
  4. Sheared LLaMA: Accelerating Language Model Pre-training via Structured PruningMengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi ChenICLR 2024 · 453 citations
  5. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model SizesCheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost et al.ACL Findings 2023 · 317 citations

Related papers

  1. Think, But Don't Tell: Implicit Reasoning for LLM-based Sequential Recommendation via Multi-Teacher DistillationWeihai Lu, Xiaoxi Cui, Chenke YinSIGIR 2026
  2. PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought DistillationTao Fan, Guoqiang Ma, Yuanfeng Song, Lixin Fan et al.EMNLP 2025 · 1 citation
  3. Distilling LLM Agent into Small Models with Retrieval and Code ToolsMinki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho et al.NeurIPS 2025 · 51 citations
  4. UniCoTT: A Unified Framework for Structural Chain-of-Thought DistillationXianwei Zhuang, Zhihong Zhu, Zhichang Wang, Xuxin Cheng et al.ICLR 2025
  5. Fine-Tuned Thoughts: Leveraging Chain-of-Thought Reasoning for Industrial Asset Health MonitoringShuxin Lin, Dhaval C. Patel, Christodoulos ConstantinidesEMNLP Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines