Attention Implements the Fisher Geometry of Exponential Families

Softmax attention normalizes scores, and Bayes’ rule normalizes log prior plus log likelihood. For finite latent symbols with exponential-family observations, we show that one attention head can implement the Bayes posterior and posterior means exactly, and that the posteriors representable by a single head are precisely log-linear. The standard exponential-family duality identity rewrites the likelihood as a negative Bregman divergence in mean/sufficient-statistic space; our attention-specific contribution is to use this identity to characterize when Bayes-aligned attention admits one globally shared quadratic metric, proving that this happens exactly when the dual potential is quadratic. When curvature varies, we give a multi-head local-curvature atlas with approximation and head-count bounds, and we extend the picture to in-context estimation through plug-in consistency, finite-sample stability, and an optimizer-agnostic converse from excess log-loss to approximate key-subspace alignment. Controlled Gaussian, Bernoulli, and Poisson ICE diagnostics illustrate these regimes, while the exact theorems remain scoped to finite discrete latent classes and suggest testable, not universal, predictions for larger learned transformers.