ACL Findings2025
Probabilistic Aggregation and Targeted Embedding Optimization for Collective Moral Reasoning in Large Language Models
Chenchen Yuan, Zheyu Zhang, Shuo Yang, Bardh Prenkaj, Gjergji Kasneci
Abstract
Large Language Models (LLMs) have shown impressive moral reasoning abilities. Yet they often diverge when confronted with complex, multi-factor moral dilemmas. To address these discrepancies, we propose a framework that synthesizes multiple LLMs' moral judgments into a collectively formulated moral judgment, realigning models that deviate significantly from this consensus. Our aggregation mechanism fuses continuous moral acceptability scores (beyond binary labels) into a collective probability, weighting contributions by model reliability. For misaligned models, a targeted embedding-optimization procedure fine-tunes token embeddings for moral philosophical theories, minimizing JS divergence to the consensus while preserving semantic integrity. Experiments on a large-scale social moral dilemma dataset show our approach builds robust consensus and improves individual model fidelity. These findings highlight the value of datadriven moral alignment across multiple models and its potential for safer, more consistent AI systems. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on13
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
- Aligning AI With Shared Human ValuesDan Hendrycks, Collin Burns, Steven Basart, Andrew Critch et al.ICLR 2021 · 878 citations
- Fast Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn et al.ICLR 2022 · 527 citations
Related papers
- Many LLMs Are More Utilitarian Than OneAnita Keshmirian, Razan Baltaji, Babak Hemmatian, Hadi Asghari et al.NeurIPS 2025 · 10 citations
- Advancing Automated Ethical Profiling in SE: a Zero-Shot Evaluation of LLM ReasoningPatrizio Migliarini, Mashal Afzal Memon, Marco Autili, Paola InverardiASE 2025
- Unveiling the Bias Impact on Symmetric Moral Consistency of Large Language ModelsZiyi Zhou, Xinwei Guo, Jiashi Gao, Xiangyu Zhao et al.NeurIPS 2024
- Diagnosing Moral Reasoning Acquisition in Language Models: Pragmatics and GeneralizationGuangliang Liu, Zimo Qi, Xitong Zhang, Lei Jiang et al.EMNLP Findings 2025
- Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMsAbhinav Rao, Aditi Khandelwal, Kumar Tanmay, Utkarsh Agarwal et al.EMNLP Findings 2023 · 32 citations
