EMNLP Findings2022
Improving Semantic Matching through Dependency-Enhanced Pre-trained Model with Adaptive Fusion
Jian Song, Di Liang, Rumei Li, Yuntao Li, Sirui Wang, Minlong Peng, Wei Wu, Yongxin Yu
Abstract
Transformer-based pre-trained models like BERT have achieved great progress on Semantic Sentence Matching. Meanwhile, dependency prior knowledge has also shown general benefits in multiple NLP tasks. However, how to efficiently integrate dependency prior structure into pre-trained models to better model complex semantic matching relations is still unsettled. In this paper, we propose the Dependency-Enhanced Adaptive Fusion Attention (DAFA), which explicitly introduces dependency structure into pre-trained models and adaptively fuses it with semantic information. Specifically, (i) DAFA first proposes a structure-sensitive paradigm to construct a dependency matrix for calibrating attention weights. (ii) It adopts an adaptive fusion module to integrate the obtained dependency information and the original semantic signals. Moreover, DAFA reconstructs the attention calculation flow and provides better interpretability. By applying it on BERT, our method achieves state-of-the-art or competitive performance on 10 public datasets, demonstrating the benefits of adaptively fusing dependency structure in semantic matching task.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance cites Question Calibration and Multi-Hop Modeling for Temporal Question Answering
- Reinforcement Learning Enhanced Muti-hop Reasoning for Temporal Knowledge Question Answering cites Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance
- Reinforcement Learning Enhanced Muti-hop Reasoning for Temporal Knowledge Question Answering cites Question Calibration and Multi-Hop Modeling for Temporal Question Answering
- Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models cites Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance
- Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models cites Question Calibration and Multi-Hop Modeling for Temporal Question Answering
- Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning cites Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance
- Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning cites Question Calibration and Multi-Hop Modeling for Temporal Question Answering
- Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks cites K-BERT: Enabling Language Representation with Knowledge Graph
- Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks cites Semantics-Aware BERT for Language Understanding
Cited by top-tier papers5
- Question Calibration and Multi-Hop Modeling for Temporal Question AnsweringChao Xue, Di Liang, Pengfei Wang, Jing ZhangAAAI 2024 · 27 citations
- Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language ModelsChao Xue, Yao Wang, Mengqiao Liu, Di Liang et al.ACL 2026 · 5 citations
- Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning PerformanceYao Wang, Di Liang, Minlong PengEMNLP 2025 · 3 citations
- Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-TuningZekai Lin, Chao Xue, Di Liang, Xingsheng Han et al.ACL 2026 · 2 citations
- Reinforcement Learning Enhanced Muti-hop Reasoning for Temporal Knowledge Question AnsweringWuzhenghong Wen, Chao Xue, Su Pan, Yuwei Sun et al.AAAI 2026
Builds on4
- K-BERT: Enabling Language Representation with Knowledge GraphWeijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang et al.AAAI 2020 · 898 citations
- Semantics-Aware BERT for Language UnderstandingZhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li et al.AAAI 2020 · 396 citations
- Self-Attention Attribution: Interpreting Information Interactions Inside TransformerYaru Hao, Li Dong, Furu Wei, Ke XuAAAI 2021 · 282 citations
- Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching TasksTingyu Xia, Yue Wang, Yuan Tian, Yi ChangWWW 2021 · 56 citations
Related papers
- Syntax-Enhanced Pre-trained ModelZenan Xu, Daya Guo, Duyu Tang, Qinliang Su et al.ACL 2021
- Lexicon Enhanced Chinese Sequence Labeling Using BERT AdapterWei Liu, Xiyan Fu, Yue Zhang, Wenming XiaoACL 2021
- Improving BERT with Syntax-aware Local AttentionZhongli Li, Qingyu Zhou, Chao Li, Ke Xu et al.ACL Findings 2021 · 33 citations
- MA-BERT: Learning Representation by Incorporating Multi-Attribute Knowledge in TransformersYou Zhang, Jin Wang, Liang-Chih Yu, Xuejie ZhangACL Findings 2021 · 23 citations
- GiLT: Augmenting Transformer Language Models with Dependency GraphsTianyu Huang, Yida Zhao, Chuyan Zhou, Kewei TuACL 2026
