On the Proper Treatment of Units in Surprisal Theory

Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units. As a result, surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices: the definition of the unit of analysis and the choice of regions of interest over which predictions are evaluated. In this paper, we disentangle these choices and give a unified framework for reasoning about surprisal over arbitrary unit inventories. We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail rather than a scientific primitive. https://github.com/samuki/ units-surprisal GPT-2 Tokens 22906 ␣don 836 't 470 ␣equal 4961 ␣words 2456 . 13 Chars T o k e n s ␣ d o n ' t ␣ e q u a l ␣ w o r d s .