EMNLP Findings2022
On the Impact of Temporal Concept Drift on Model Explanations
Zhixue Zhao, George Chrysostomou, Kalina Bontcheva, Nikolaos Aletras
Abstract
Explanation faithfulness of model predictions in natural language processing is typically evaluated on held-out data from the same temporal distribution as the training data (i.e. synchronous settings). While model performance often deteriorates due to temporal variation (i.e. temporal concept drift), it is currently unknown how explanation faithfulness is impacted when the time span of the target data is different from the data used to train the model (i.e. asynchronous settings). For this purpose, we examine the impact of temporal variation on model explanations extracted by eight feature attribution methods and three select-then-predict models across six text classification tasks. Our experiments show that (i) faithfulness is not consistent under temporal variations across feature attribution methods (e.g. it decreases or increases depending on the method), with an attention-based method demonstrating the most robust faithfulness scores across datasets; and (ii) select-then-predict models are mostly robust in asynchronous settings with only small degradation in predictive performance. Finally, feature attribution methods show conflicting behavior when used in FRESH (i.e. a selectand-predict model) and for measuring sufficiency/comprehensiveness (i.e. as post-hoc methods), suggesting that we need more robust metrics to evaluate post-hoc explanation faithfulness. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Mind the Gap: Assessing Temporal Generalization in Neural Language Models cites Temporally-Informed Analysis of Named Entity Recognition
- ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks cites Mind the Gap: Assessing Temporal Generalization in Neural Language Models
- ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks cites Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media
- ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks cites Temporally-Informed Analysis of Named Entity Recognition
- LexTempus: Enhancing Temporal Generalizability of Legal Language Models Through Dynamic Mixture of Experts cites Mind the Gap: Assessing Temporal Generalization in Neural Language Models
- LexTempus: Enhancing Temporal Generalizability of Legal Language Models Through Dynamic Mixture of Experts cites ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks
- LexTempus: Enhancing Temporal Generalizability of Legal Language Models Through Dynamic Mixture of Experts cites Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media
- LexTempus: Enhancing Temporal Generalizability of Legal Language Models Through Dynamic Mixture of Experts cites Temporally-Informed Analysis of Named Entity Recognition
- RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning cites Incorporating Attribution Importance for Improving Faithfulness Metrics
- Incorporating Attribution Importance for Improving Faithfulness Metrics cites A Diagnostic Study of Explainability Techniques for Text Classification
Cited by top-tier papers4
- Incorporating Attribution Importance for Improving Faithfulness MetricsZhixue Zhao, Nikolaos AletrasACL 2023 · 4 citations
- LexTempus: Enhancing Temporal Generalizability of Legal Language Models Through Dynamic Mixture of ExpertsT. Y. S. S. Santosh, Tuan-Quang VuongACL 2025 · 1 citation
- ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification TasksT. Y. S. S. Santosh, Tuan-Quang Vuong, Matthias GrabmairACL 2024
- RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like ReasoningJason Chan, Robert J. Gaizauskas, Zhixue ZhaoICML 2025
Builds on11
- Mind the Gap: Assessing Temporal Generalization in Neural Language ModelsAngeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal et al.NeurIPS 2021 · 315 citations
- A Diagnostic Study of Explainability Techniques for Text ClassificationPepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle AugensteinEMNLP 2020 · 158 citations
- Analysing Lexical Semantic Change with Contextualised Word RepresentationsMario Giulianelli, Marco Del Tredici, Raquel FernándezACL 2020 · 118 citations
- Temporally-Informed Analysis of Named Entity RecognitionShruti Rijhwani, Daniel Preotiuc-PietroACL 2020 · 49 citations
- Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social MediaPaul Röttger, Janet B. PierrehumbertEMNLP Findings 2021 · 47 citations
Related papers
- An Empirical Study on Explanations in Out-of-Domain SettingsGeorge Chrysostomou, Nikolaos AletrasACL 2022
- "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text ClassificationJasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm et al.EMNLP 2022 · 29 citations
- Provably Better Explanations with Optimized Aggregation of Feature AttributionsThomas Decker, Ananta R. Bhattarai, Jindong Gu, Volker Tresp et al.ICML 2024 · 7 citations
- Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text ClassificationGeorge Chrysostomou, Nikolaos AletrasACL 2021
- A Causal Lens for Evaluating Faithfulness MetricsKerem Zaman, Shashank SrivastavaEMNLP 2025
