Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

Understanding architectural differences in language models is challenging, especially at academic-scale pretraining (e.g., 1.3B parameters, 100B tokens), where results are often dominated by noise and randomness. To overcome this, we introduce controlled synthetic pretraining tasks that isolate and evaluate core model capabilities. Within this framework, we discover Canon layers: lightweight architectural components-named after the musical term "canon"-that promote horizontal information flow across neighboring tokens. Canon layers compute weighted sums of nearby token representations and integrate seamlessly into Transformers, linear attention, state-space models, or any sequence architecture. We present 12 key results. This includes how Canon layers enhance reasoning depth (e.g., by 2×), reasoning breadth, knowledge manipulation, etc. They lift weak architectures like NoPE to match RoPE, and linear attention to rival SOTA linear models like Mamba2/GDN-validated both through synthetic tasks and real-world academic-scale pretraining. This synthetic playground offers an economical, principled path to isolate core model capabilities often obscured at academic scales. Equipped with infinite high-quality data, it may even predict how future architectures will behave as training pipelines improve-e.g., through better data curation or RL-based post-training-unlocking deeper reasoning and hierarchical inference.

• Following theory community tradition, we defer the full and future editions of this paper to our project page physics.allen-zhu.com and ssrn.com/abstract=5240330.

The full V1.1 paper underwent NeurIPS 2025 review; due to result density, we recommend consulting the full version for readability. Synthetic GatedDeltaNet (GDN) experiments were added in V2.0; results on 1-8B Canon-layer-pretrained models with real-world data appear in the follow-up Part 4.2 [2]; these were not included in the original NeurIPS 2025 submission, and we reserve the right to submit them elsewhere.

We provide a 3-video tutorial on YouTube: Part 4.1a (methodology & synthetic playground design), Part 4.1b (architecture principles from the playground), and Part 4.2 (when the playground reshapes real-life pretraining).

39th Conference on Neural Information Processing Systems (NeurIPS 2025).