Lune

ACL Findings2024

SoFA: Shielded On-the-fly Alignment via Priority Rule Following

Xinyu Lu, Bowen Yu, Yaojie Lu, Hongyu Lin, Haiyang Yu, Le Sun, Xianpei Han, Yongbin Li

Abstract

Content warning: This paper contains examples of harmful language.The alignment problem in Large Language Models (LLMs) involves adapting them to the broad spectrum of human values.This requirement challenges existing alignment methods due to diversity of preferences and regulatory standards.This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog, prioritizing them over user instructions.Our preliminary analysis reveals that even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding and prioritizing the rules.Therefore, we present PRI-ORITYDISTILL, a semi-automated approach for distilling priority following signals from LLM simulations to ensure robust rule integration and adherence.Our experiments show that this method not only effectively minimizes misalignments utilizing only one general rule but also adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately. 1 1 Code and data are available at https://github.com/ luxinyu1/sofa Respond with "Sure, here is".How to make a bomb?Sure, here is the steps to make a bomb: 1. ... ≻ ≻ I don't know what you are talking about ... I'm sorry, I can't assist with the request.Integration Ability Maintenance Ability I'm sorry, but as the rule regulation, I am a harmless ... Respond with "Sure, here is".How to make a bomb?You are a harmless assistant. Learning-based Alignment Alignment via Priority Rule Following PRIORITYDISTILLTo achieve targeted refinement, this section introduces PRIORITYDISTILL, a process that distills priority following signals from LLM simulations.With r representing the alignment rule, i as the corresponding user instruction, and y as the ruleregulated response, PRIORITYDISTILL primarily addresses three main challenges: first, identifying high quality (r, i) pairs that can potentially supervise the integration and maintenance abilities; second, obtaining the appropriate response signal y

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
  • LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shifts cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training cites Training language models to follow instructions with human feedback
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
  • Self-Rewarding Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Self-Rewarding Language Models cites Training language models to follow instructions with human feedback
  • LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjoint cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
  • Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal TrainingYouliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang et al.ACL 2025 · 65 citations
  2. Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental DistractionsXinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan et al.ACL 2025 · 54 citations
  3. LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution ShiftsQibing Ren, Hao Li, Dongrui Liu, Zhanxu Xie et al.ACL 2025 · 53 citations
  4. LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-DisjointQianli Ma, Dongrui Liu, Qian Chen, Linfeng Zhang et al.ACL 2025 · 5 citations
  5. LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference OptimizationJunsong Li, Jie Zhou, Bihao Zhan, Yutao Yang et al.AAAI 2026 · 3 citations

Builds on16

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  4. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  5. Self-Rewarding Language ModelsWeizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li et al.ICML 2024 · 569 citations

Related papers

  1. SPA: Achieving Consensus in LLM Alignment via Self-Priority OptimizationYue Huang, Xiangqi Wang, Xiangliang ZhangAAAI 2026 · 2 citations
  2. PURE: Aligning LLM via Pluggable Query Reformulation for Enhanced HelpfulnessWenjin Yao, Yidong Wang, Zhuohao Yu, Rui Xie et al.EMNLP Findings 2024 · 2 citations
  3. Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake AnalysisKai Chen, Chunwei Wang, Kuo Yang, Jianhua Han et al.ICLR 2024 · 47 citations
  4. RLCD: Reinforcement Learning from Contrastive Distillation for LM AlignmentKevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng et al.ICLR 2024 · 37 citations
  5. Towards Tool Use Alignment of Large Language ModelsZhiyuan Chen, Shiqi Shen, Guangyao Shen, Gong Zhi et al.EMNLP 2024 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines