Improving Reasoning Capabilities in Small Models through Mixture-of-layers Distillation with Stepwise Attention on Key Information

The significant computational demands of large language models have increased interest in distilling reasoning abilities into smaller models via Chain-of-Thought (CoT) distillation. Current CoT distillation methods mainly focus on transferring teacher-generated rationales for complex reasoning to student models. However, they do not adequately explore teachers' dynamic attention toward critical information during reasoning. We find that language models exhibit progressive attention shifts towards key information during reasoning, which implies essential clues for drawing conclusions. Building on this observation and analysis, we introduce a novel CoT distillation framework that transfers the teacher's stepwise attention on key information to the student model. This establishes structured guidance for the student's progressive concentration on key information during reasoning. More importantly, we develop a Mixture of Layers module enabling dynamic alignment that adapts to different layers between the teacher and student. Our method achieves consistent performance improvements across multiple mathematical and commonsense reasoning datasets. To our knowledge, it is the first method to leverage stepwise attention within CoT distillation to improve small model reasoning. Question: (Step1)A mailman has to give 4 pieces of junk mail to each house in each of the 16 blocks. If there are 17 houses in each block, How many pieces of junk mail should he give in total? Rationale: (Step2)The mailman has to give 4 pieces of junk mail to each house in each of the 16 blocks. (Step3)There are 17 houses in each block. (Step4)So, the total number of houses is 16 * 17 = 272 houses. (Step5)The mailman has to give 4 pieces of junk mail to each house, so the total number of junk mail pieces is 272 * 4 = 1088. Answer: (16 * 17 * 4 ) Question Answer