PRISM: Probabilistic Reward Model with Inherent Structural Modeling

Standard evaluators, such as reward models, compress diverse human judgments into a single scalar, conflating valid Subjective Preference with Epistemic Uncertainty. This structural mismatch often leads to brittle alignment and reward hacking. To address this, we pro-pose PRISM which reinterprets reward evaluation as a conditional distribution parameterized by a Mixture of Gaussians(MOG). PRISM structurally disentangles these factors: distinct Gaussian experts emerge to capture conflicting preference dimensions, while their variance estimates quantify uncertainty, acting as a dynamic reliability gate during optimization. We introduce a two-stage training strategy to learn these disentangled representations from scalable pairwise comparisons without requiring massive fine-grained annotations. Empirical results show that PRISM significantly outperforms scalar baselines in both accuracy and generalization. Furthermore, in downstream Rubric-based Reinforcement Learning, PRISM effectively mitigates reward hacking, yielding policies that are more robust and resilient to distribution shifts.