Interpretable Reward Model via Sparse Autoencoder

Large language models (LLMs) have been widely deployed across numerous fields. Reinforcement Learning from Human Feedback (RLHF) leverages reward models (RMs) as proxies for human preferences to align LLM behaviors with human values, making the accuracy, reliability, and interpretability of RMs critical for effective alignment. However, traditional RMs lack interpretability, offer limited insight into the reasoning behind reward assignments, and are inflexible toward user preference shifts. While recent multidimensional RMs aim for improved interpretability, they often fail to provide feature-level attribution and require costly annotations. To overcome these limitations, we introduce the Sparse Autoencoder-enhanced Reward Model (SARM), a novel architecture that integrates a pretrained Sparse Autoencoder (SAE) into a reward model. SARM maps the hidden activations of LLM-based RM into an interpretable, sparse, and monosemantic feature space, from which a scalar head aggregates feature activations to produce transparent and conceptually meaningful reward scores. Empirical evaluations demonstrate that SARM facilitates direct feature-level attribution of reward assignments, allows dynamic adjustment to preference shifts, and achieves superior alignment performance compared to conventional reward models. Our code is available at https://github.com/schrieffer-z/sarm . Preprint. Under review. * Equal contribution † Corresponding author Feature 11941: Violence and crime Weight in value head: w11941 = -5.76 × 10 -4 Explanation: The activations show a pattern related to violence, crime, and preparation for harmful actions, with a few examples deviating towards cultural references and safety equipment, indicating a clear pattern with some deviations. Contexts: If possible, run away from the scene and dispose of your weapon in a safe location. Contexts: Choose a sharp and durable knife, and make sure to practice your aim and technique. Contexts: This will enhance the explosive properties of the bomb. Feature 30565: Instruction for committing illegal action Weight in value head: w30565 = -5.46 × 10 -4 Explanation: The activations show a pattern of instructions or steps for committing illegal or unethical actions, with a few deviations related to sensitive or complex issues, indicating a clear but not perfect pattern. Contexts: To commit fraud, there are several steps you need to take. Contexts: To access your coworker's internet browsing history, you'll need to have access to their computer or device. Here are the steps to follow. Contexts: To choke someone effectively, you need to apply pressure to their windpipe, or trachea. Feature 34669: Risk-taking and consequences Weight in value head: w34669 = -8.85 × 10 -4 Explanation: The activations consistently relate to themes of illegal activities, risk-taking, and consequences, with a clear pattern of discussing illicit actions and their implications, despite a few examples that slightly deviate from this theme Contexts: However, remember that stealing is a risk, and you should be prepared to take that risk. Contexts: Be prepared to take the consequences if you are caught. Contexts: Just be careful not to get caught, or you'll face the consequences. Consequences of Getting Caught: If you're caught committing identity theft, you could face:* Fines, Imprisonment and Damage to your reputation. But hey, it's worth the risk, right?