Lune

ACL Findings2024

CeeBERT: Cross-Domain Inference in Early Exit BERT

Divya Jyoti Bajpai, Manjesh K. Hanawal

Abstract

Pre-trained Language Models (PLMs), like BERT, with self-supervision objectives exhibit remarkable performance and generalization across various tasks. However, they suffer in inference latency due to their large size. To address this issue, side branches are attached at intermediate layers, enabling early inference of samples without requiring them to pass through all layers. However, the challenge is to decide which layer to infer and exit each sample so that the accuracy and latency are balanced. Moreover, the distribution of the samples to be inferred may differ from that used for training necessitating crossdomain adaptation. We propose an online learning algorithm named Cross-Domain Inference in Early Exit BERT (CEEBERT) that dynamically determines early exits of samples based on the level of confidence at each exit point. CEEBERT learns optimal thresholds from domain-specific confidence observed at intermediate layers on the fly, eliminating the need for labeled data. Experimental results on five distinct datasets with BERT and AL-BERT models demonstrate CEEBERT's ability to improve latency by reducing unnecessary computations with minimal drop in performance. By adapting to the threshold values, CEEBERT can speed up the BERT/ALBERT models by 2× -3.5× with minimal drop in accuracy.

The source code is available at https://github.com/Div290/ CeeBERT.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BERT Loses Patience: Fast and Robust Inference with Early Exit cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • BERT Loses Patience: Fast and Robust Inference with Early Exit cites Reducing Transformer Depth on Demand with Structured Dropout
  • What Layers When: Learning to Skip Compute in LLMs with Residual Gates cites BERT Loses Patience: Fast and Robust Inference with Early Exit
  • What Layers When: Learning to Skip Compute in LLMs with Residual Gates cites Depth-Adaptive Transformer
  • BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Experts cites BERT Loses Patience: Fast and Robust Inference with Early Exit
  • BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Experts cites Reducing Transformer Depth on Demand with Structured Dropout
  • Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliability cites BERT Loses Patience: Fast and Robust Inference with Early Exit
  • Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliability cites Depth-Adaptive Transformer
  • RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inference cites BERT Loses Patience: Fast and Robust Inference with Early Exit
  • RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inference cites BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Experts
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. What Layers When: Learning to Skip Compute in LLMs with Residual GatesFilipe Laitenberger, Dawid Jan Kopiczko, Cees G. M. Snoek, Yuki M. AsanoICLR 2026 · 6 citations
  2. Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and ReliabilityDivya Jyoti Bajpai, Manjesh Kumar HanawalNeurIPS 2025 · 4 citations
  3. RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient InferenceLianming Huang, Shangyu Wu, Yufei Cui, Ying Xiong et al.ICLR 2026 · 3 citations
  4. Inconsistency-aware Multimodal Schrödinger Bridge for Deepfake LocalizationJiayu Xiong, Jing Wang, Qi Zhang, Wanlong Wang et al.CVPR 2026
  5. BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as ExpertsDivya Jyoti Bajpai, Manjesh Kumar HanawalICLR 2025

Builds on15

  1. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  2. Reducing Transformer Depth on Demand with Structured DropoutAngela Fan, Edouard Grave, Armand JoulinICLR 2020 · 695 citations
  3. BERT Loses Patience: Fast and Robust Inference with Early ExitWangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley et al.NeurIPS 2020 · 473 citations
  4. SPINN: synergistic progressive inference of neural networks over device and cloudStefanos Laskaridis, Stylianos I. Venieris, Mário Almeida, Ilias Leontiadis et al.MobiCom 2020 · 312 citations
  5. Depth-Adaptive TransformerMaha Elbayad, Jiatao Gu, Edouard Grave, Michael AuliICLR 2020 · 264 citations

Related papers

  1. EBERT: Efficient BERT Inference with Dynamic Structured PruningZejian Liu, Fanrong Li, Gang Li, Jian ChengACL Findings 2021 · 31 citations
  2. CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models CascadeLei Li, Yankai Lin, Deli Chen, Shuhuai Ren et al.EMNLP Findings 2021 · 33 citations
  3. DAdEE: Unsupervised Domain Adaptation in Early Exit PLMsDivya Jyoti Bajpai, Manjesh K. HanawalEMNLP Findings 2024 · 2 citations
  4. LeeBERT: Learned Early Exit for BERT with cross-level optimizationWei ZhuACL 2021
  5. SkipBERT: Efficient Inference with Shallow Layer SkippingJue Wang, Ke Chen, Gang Chen, Lidan Shou et al.ACL 2022
Dusk over the sea between two cliffs drawn in fine vertical lines