The Burden of Interactive Alignment with Inconsistent Preferences

From media platforms to chatbots, algorithms shape how people interact, learn, and discover information. Such interactions between users and an algorithm often unfold over multiple steps, during which strategic users can guide the algorithm to better align with their true interests by selectively engaging with content. However, users frequently exhibit inconsistent preferences: they may spend considerable time on content that offers little long-term value, inadvertently signaling that such content is desirable. Focusing on the user side, this raises a key question: what does it take for such users to align the algorithm with their true interests? To investigate these dynamics, we model the user's decision process as split between a rational "system 2" that decides whether to engage and an impulsive "system 1" that determines how long engagement lasts. We then study a multi-leader, singlefollower extensive Stackelberg game, where users, specifically system 2, lead by committing to engagement strategies and the algorithm best-responds based on observed interactions. We define the burden of alignment as the minimum horizon over which users must optimize to effectively steer the algorithm. We show that a critical horizon exists: users who are sufficiently foresighted can achieve alignment, while those who are not are instead aligned to the algorithm's objective. This critical horizon can be long, imposing a substantial burden. However, even a small, costly signal (e.g., an extra click) can significantly reduce it. Overall, our framework explains how users with inconsistent preferences can align an engagement-driven algorithm with their interests in a Stackelberg equilibrium, highlighting both the challenges and potential remedies for achieving alignment.

Our setting and model. When users have consistent preferences-i.e., the engagement length is in proportion to user's true reward-the alignment problem reduces to engagement maximization. Recent advances in designing instruction-following language models rely on this assumption of consistent preferences, often using models such as Bradley-Terry [1] to directly or indirectly infer human rewards and optimize them to maximize human approval [2][3][4]. Similarly, recommender systems are typically designed to optimize recommendations that maximize user engagement [5].

Users can, however, have inconsistent preferences, where their actions may not reflect their true interests. This often occurs when a user's decision results from a combination of impulsive system 1 and rational system 2 processes [6-8], or when consumption choices are influenced by both long-term benefits (enrichment) and a desire for instant gratification (temptation) [9]. In such cases, revealed preferences do not necessarily align with the user's true preferences.

39th Conference on Neural Information Processing Systems (NeurIPS 2025).