The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training

Self-attention is essential to Transformer architectures, yet how information is embedded in the selfattention matrices and how different objective functions impact this process remains unclear. We present a mathematical framework to analyze self-attention matrices by deriving the structures governing their weight updates. Using this framework, we demonstrate that bidirectional training induces symmetry in the weight matrices, while autoregressive training results in directionality and column dominance. Our theoretical findings are validated across multiple Transformer models -including ModernBERT, GPT, LLaMA3, and Mistral -and input modalities like text, vision, and audio. Finally, we apply these insights by showing that symmetric initialization improves the performance of encoder-only models on language tasks. This mathematical analysis offers a novel theoretical perspective on how information is embedded through self-attention, thereby improving the interpretability of Transformer models. We provide proof for this proposition with related remarks in Appendix S1.5.1. Note that all mathematical results derived so far rely solely on the structure of self-attention and make no assumptions about the input data. In the next section, we build on these results to link structural patterns in W qk with the specific form of the objective function. The relation between objective functions and structures in self-attention matrices In this final section, we show how to relate these properties to the specific objective function. Crucially, the number of times a token t * appears as context or as a prediction depends on the training objective. Autoregressive training implicitly introduces