Lune

EMNLP Findings2024

SEAVER: Attention Reallocation for Mitigating Distractions in Language Models for Conditional Semantic Textual Similarity Measurement

Baixuan Li, Yunlong Fan, Zhiqiang Gao

Abstract

Conditional Semantic Textual Similarity (C-STS) introduces specific limiting conditions to the traditional Semantic Textual Similarity (STS) task, posing challenges for STS models. Language models employing cross-encoding demonstrate satisfactory performance in STS, yet their effectiveness significantly diminishes in C-STS. In this work, we argue that the failure is due to the fact that the redundant information in the text distracts language models from the required condition-relevant information. To alleviate this, we propose SElf-Augmentation Via SElf-Reweighting (SEAVER), which, based solely on models' internal attention and without the need for external auxiliary information, adaptively reallocates the model's attention weights by emphasizing the importance of condition-relevant tokens. On the C-STS-2023 test set, SEAVER consistently improves performance of all million-scale fine-tuning baseline models (up to around 3 points), and even surpasses performance of billion-scale fewshot prompted large language models (such as GPT-4). Our code is available at https: //github.com/BaixuanLi/SEAVER.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • PoLi-RL: A Point-to-List Reinforcement Learning Framework for Conditional Semantic Textual Similarity cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
  • Perceiver IO: A General Architecture for Structured Inputs & Outputs cites Language Models are Few-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. PoLi-RL: A Point-to-List Reinforcement Learning Framework for Conditional Semantic Textual SimilarityZixin Song, Bowen Zhang, Qian-Wen Zhang, Di Yin et al.ICLR 2026 · 1 citation

Builds on9

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  3. SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
  4. Perceiver IO: A General Architecture for Structured Inputs & OutputsAndrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch et al.ICLR 2022 · 797 citations
  5. Revisiting Few-sample BERT Fine-tuningTianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger et al.ICLR 2021 · 172 citations

Related papers

  1. Revisiting Self-training for Few-shot Learning of Language ModelYiming Chen, Yan Zhang, Chen Zhang, Grandee Lee et al.EMNLP 2021 · 35 citations
  2. STraTA: Self-Training with Task Augmentation for Better Few-shot LearningTu Vu, Minh-Thang Luong, Quoc V. Le, Grady Simon et al.EMNLP 2021 · 25 citations
  3. Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMsZifeng Cheng, Lingyun Qian, Zhiwei Jiang, Cong Wang et al.ACL 2026
  4. Enhancing Transformer-based Semantic Matching for Few-shot Learning through Weakly Contrastive Pre-trainingWei Yang, Tengfei Huo, Zhiqiang LiuACM MM 2024
  5. USE: A Unified Self-Ensembling Framework for Test-Time Prompt TuningSiru Jiang, Jian Liang, Ran He, Tieniu TanICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines