SenseJudge: Human-Centric Preference-Driven Judgment Framework

Using Large Language Models (LLMs) as judges across scenarios such as modelresponse assessment is becoming an increasingly accepted paradigm. However, existing judgment approaches often rely on trained judgers using fixed preference data, which tend to overlook diverse user preferences and struggle to adapt to real-world human-AI dialogue scenarios. To address these limitations, we propose SenseJudge, a customizable judgment framework driven by human preferences, and SenseBench, a diverse and challenging instruction-following benchmark derived from real-world multi-turn interactions. We evaluate the framework and benchmark on two tasks: 1) LLMs as personalized judges, and 2) model ranking. Extensive experiments show that the SenseJudge framework outperforms the compared judgment methods and models in the LLMs-as-personalized-judges task and produces model rankings that align with human judgments. Additionally, we conducted analyses on position bias and consistency, alongside ablation studies, which support the robustness of SenseJudge. Our data and code are publicly available in the GitHub repository.