Lune

ACL Findings2022

Extracting Latent Steering Vectors from Pretrained Language Models

Nishant Subramani, Nivedita Suresh, Matthew E. Peters

Abstract

Prior work on controllable text generation has focused on learning how to control language models through trainable decoding, smartprompt design, or fine-tuning based on a desired objective. We hypothesize that the information needed to steer the model to generate a target sentence is already encoded within the model. Accordingly, we explore a different approach altogether: extracting latent vectors directly from pretrained language model decoders without fine-tuning. Experiments show that there exist steering vectors, which, when added to the hidden states of the language model, generate a target sentence nearly perfectly (> 99 BLEU) for English sentences from a variety of domains. We show that vector arithmetic can be used for unsupervised sentiment transfer on the Yelp sentiment benchmark, with performance comparable to models tailored to this task. We find that distances between steering vectors reflect sentence similarity when evaluated on a textual similarity benchmark (STS-B), outperforming pooled hidden states of models. Finally, we present an analysis of the intrinsic properties of the steering vectors. Taken together, our results suggest that frozen LMs can be effectively controlled through their latent steering space. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

108 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Function Vectors in Large Language Models cites Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
  • Function Vectors in Large Language Models cites Language Models are Few-Shot Learners
  • Implicit In-context Learning cites Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
  • Implicit In-context Learning cites Language Models are Few-Shot Learners
  • Many-shot Jailbreaking cites Language Models are Few-Shot Learners
  • ReFT: Representation Finetuning for Language Models cites Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
  • GeDi: Generative Discriminator Guided Sequence Generation cites Plug and Play Language Models: A Simple Approach to Controlled Text Generation
  • GeDi: Generative Discriminator Guided Sequence Generation cites Language Models are Few-Shot Learners
  • Inference-Time Intervention: Eliciting Truthful Answers from a Language Model cites Plug and Play Language Models: A Simple Approach to Controlled Text Generation
  • Inference-Time Intervention: Eliciting Truthful Answers from a Language Model cites GeDi: Generative Discriminator Guided Sequence Generation
  • Inference-Time Intervention: Eliciting Truthful Answers from a Language Model cites Language Models are Few-Shot Learners
  • AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers108

  1. Implicit In-context LearningZhuowei Li, Zihao Xu, Ligong Han, Yunhe Gao et al.ICLR 2025 · 1,989 citations
  2. Inference-Time Intervention: Eliciting Truthful Answers from a Language ModelKenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister et al.NeurIPS 2023 · 1,549 citations
  3. Many-shot JailbreakingCem Anil, Esin Durmus, Nina Panickssery, Mrinank Sharma et al.NeurIPS 2024 · 338 citations
  4. ReFT: Representation Finetuning for Language ModelsZhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger et al.NeurIPS 2024 · 233 citations
  5. Function Vectors in Large Language ModelsEric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller et al.ICLR 2024 · 229 citations

Builds on12

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Plug and Play Language Models: A Simple Approach to Controlled Text GenerationSumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung et al.ICLR 2020 · 1,166 citations
  3. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  4. GeDi: Generative Discriminator Guided Sequence GenerationBen Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar et al.EMNLP Findings 2021 · 219 citations
  5. Residual Energy-Based Models for Text GenerationYuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam et al.ICLR 2020 · 147 citations

Related papers

  1. Word Embeddings Are Steers for Language ModelsChi Han, Jialiang Xu, Manling Li, Yi Fung et al.ACL 2024 · 8 citations
  2. SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language ModelsZirui He, Mingyu Jin, Bo Shen, Ali Payani et al.EMNLP 2025 · 1 citation
  3. Attribute Alignment: Controlling Text Generation from Pre-trained Language ModelsDian Yu, Zhou Yu, Kenji SagaeEMNLP Findings 2021 · 26 citations
  4. TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted RestylingParker Riley, Noah Constant, Mandy Guo, Girish Kumar et al.ACL 2021
  5. Improving Instruction-Following in Language Models through Activation SteeringAlessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines