Differentially Private Steering for Large Language Model Alignment
Anmol Goel, Yaxi Hu, Iryna Gurevych, Amartya Sanyal
Abstract
Aligning Large Language Models (LLMs) with human values and away from undesirable behaviors (such as hallucination) has become increasingly important. Recently, steering LLMs towards a desired behavior via activation editing has emerged as an effective method to mitigate harmful generations at inference-time. Activation editing modifies LLM representations by preserving information from positive demonstrations (e.g., truthful) and minimising information from negative demonstrations (e.g., hallucinations). When these demonstrations come from a private dataset, the aligned LLM may leak private information contained in those private samples. In this work, we present the first study of aligning LLM behavior with private datasets. Our work proposes the Private Steering for LLM Alignment (PSA) algorithm to edit LLM activations with differential privacy (DP) guarantees. We conduct extensive experiments on seven different benchmarks with opensource LLMs of different sizes (0.5B to 7B) and model families (LlaMa, Qwen, Mistral and Gemma). Our results show that PSA achieves DP guarantees for LLM alignment with minimal loss in performance, including alignment metrics, openended text generation quality, and general-purpose reasoning. We also develop the first Membership Inference Attack (MIA) for evaluating and auditing the empirical privacy for the problem of LLM steering via activation editing. Our experiments support the theoretical guarantees by showing improved guarantees for our PSA algorithm compared to several existing non-private techniques. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Steering When Necessary: Flexible Steering Large Language Models with Backtracking cites Measuring Massive Multitask Language Understanding
- Steering When Necessary: Flexible Steering Large Language Models with Backtracking cites Finetuned Language Models are Zero-Shot Learners
- PrivGate: Steering Contextual Integrity in LLMs via Latent Space Geometry cites Extracting Training Data from Large Language Models
- Private Direct Preference Optimization for LLM Alignment cites Deep Learning with Differential Privacy
- Private Direct Preference Optimization for LLM Alignment cites Training language models to follow instructions with human feedback
- ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment cites Measuring Massive Multitask Language Understanding
- Finetuned Language Models are Zero-Shot Learners cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Finetuned Language Models are Zero-Shot Learners
- Training language models to follow instructions with human feedback cites Extracting Training Data from Large Language Models
- Extracting Training Data from Large Language Models cites Deep Learning with Differential Privacy
Cited by top-tier papers5
- ODESteer: A Unified ODE-Based Steering Framework for LLM AlignmentHongjue Zhao, Haosen Sun, Jiangtao Kong, Xiaochang Li et al.ICLR 2026 · 16 citations
- Steering When Necessary: Flexible Steering Large Language Models with BacktrackingZifeng Cheng, Jinwei Gan, Zhiwei Jiang, Cong Wang et al.NeurIPS 2025 · 9 citations
- Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning ProcessZhenyu Zhang, Shujian Zhang, John Lambert, Wenxuan Zhou et al.ICML 2026 · 3 citations
- PrivGate: Steering Contextual Integrity in LLMs via Latent Space GeometryRunshan Hu, Yukun Dong, Yingying Huangfu, Ruohan Zhao et al.ICML 2026
- Private Direct Preference Optimization for LLM AlignmentYangfan Jiang, Fei Wei, Ergute Bao, Xiaokui Xiao et al.CCS 2026
Builds on39
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Deep Learning with Differential PrivacyMartín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan et al.CCS 2016 · 7,620 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- Extracting Training Data from Large Language ModelsNicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski et al.USENIX Security 2021 · 2,866 citations
Related papers
- Spectral Editing of Activations for Large Language Model AlignmentYifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen et al.NeurIPS 2024 · 66 citations
- Jailbreak Open-Sourced Large Language Models via Enforced DecodingHangfan Zhang, Zhimeng Guo, Huaisheng Zhu, Bochuan Cao et al.ACL 2024
- PaCE: Parsimonious Concept Engineering for Large Language ModelsJinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Darshan Thaker et al.NeurIPS 2024 · 20 citations
- Steering Llama 2 via Contrastive Activation AdditionNina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong et al.ACL 2024
- PrivSV: Differentially Private Steering Vector for Large Language ModelsHaocheng Yang, Xiang Cheng, Chenhao Sun, Pengfei Zhang et al.AAAI 2026
