Rotary Position Encodings for Graphs

We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transform ers (ViTs), can be applied to graphstructured data. We find that rotating tokens depending on the spectrum of the graph Laplacian efficiently injects structural information into the attention mechanism, boosting performance in synthetic and realworld graph learning tasks. This ap proach, coined Wave-Induced Rotary Encodings (WIRE), enjoys intriguing theoretical properties: it recovers regular RoPE on grids, and depends asymptotically on the graph effective resistance. Unlike biasbased relative position encodings, WIRE is compatible with linear attention. 1 For legibility, we generally suppress the dependence of RoPE(𝒓 𝑖 ) on 𝝎 𝑛 𝑑/2 𝑛=1 , leaving it implicit. 2 Given this property, some researchers taxonomise RoPE as a type of relative position encoding (RPE). However, we prefer to distinguish it as a separate class of PE, since PEs based on other high dimensional rotations in SO(𝑑) are not necessarily translationally invariant (Schenck et al., 2025) .