Lune

EMNLP Findings2023

Zero-Shot-BERT-Adapters: a Zero-Shot Pipeline for Unknown Intent Detection

Daniele Comi, Dimitrios Christofidellis, Pier Francesco Piazza, Matteo Manica

Abstract

Intent discovery is a crucial task in natural language processing, and it is increasingly relevant for various of industrial applications. Identifying novel, unseen intents from user inputs remains one of the biggest challenges in this field. Herein, we propose Zero-Shot-BERT-Adapters, a two-stage method for multilingual intent discovery relying on a Transformer architecture fine-tuned with Adapters. We train the model for Natural Language Inference (NLI) and later perform unknown intent classification in a zeroshot setting for multiple languages. In our evaluation, we first analyze the quality of the model after adaptive fine-tuning on known classes. Secondly, we evaluate its performance in casting intent classification as an NLI task. Lastly, we test the zero-shot performance of the model on unseen classes, showing how Zero-Shot-BERT-Adapters can effectively perform intent discovery by generating semantically similar intents, if not equal, to the ground-truth ones. Our experiments show how Zero-Shot-BERT-Adapters outperforms various baselines in two zero-shot settings: known intent classification and unseen intent discovery. The proposed pipeline holds the potential for broad application in customer care. It enables automated dynamic triage using a lightweight model that can be easily deployed and scaled in various business scenarios, unlike large language models. Zero-Shot-BERT-Adapters represents an innovative multi-language approach for intent discovery, enabling the online generation of novel intents. A Python package implementing the pipeline and the new datasets we compiled are available at the following link: https://github. com/GT4SD/zero-shot-bert-adapters.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Language Models are Few-Shot Learners
  • Training language models to follow instructions with human feedback cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on7

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
  4. Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue SystemsZihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu et al.AAAI 2020 · 105 citations
  5. Unknown Intent Detection Using Gaussian Mixture Model with an Application to Zero-shot Intent ClassificationGuangfeng Yan, Lu Fan, Qimai Li, Han Liu et al.ACL 2020 · 69 citations

Related papers

  1. ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language AdaptersVipul Rathore, Rajdeep Dhingra, Parag Singla, MausamEMNLP 2023
  2. Effectiveness of Pre-training for Few-shot Intent ClassificationHaode Zhang, Yuwei Zhang, Li-Ming Zhan, Jiaxin Chen et al.EMNLP Findings 2021 · 40 citations
  3. Cross-Lingual Transfer in Zero-Shot Cross-Language Entity LinkingElliot Schumacher, James Mayfield, Mark DredzeACL Findings 2021 · 4 citations
  4. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual TransferJonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian RuderEMNLP 2020 · 36 citations
  5. MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual TransferAlan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder et al.EMNLP Findings 2021 · 57 citations
Dusk over the sea between two cliffs drawn in fine vertical lines