EMNLP Findings2020
Optimizing BERT for Unlabeled Text-Based Items Similarity
Itzik Malkiel, Oren Barkan, Avi Caciularu, Noam Razin, Ori Katz, Noam Koenigstein
Abstract
Language models that utilize extensive selfsupervised pre-training from unlabeled text, have recently shown to significantly advance the state-of-the-art performance in a variety of language understanding tasks. However, it is yet unclear if and how these recent models can be harnessed for conducting text-based recommendations. In this work, we introduce RecoBERT, a BERT-based approach for learning catalog-specialized language models for text-based item recommendations. We suggest novel training and inference procedures for scoring similarities between pairs of items, that don't require item similarity labels. Both the training and the inference techniques were designed to utilize the unlabeled structure of textual catalogs, and minimize the discrepancy between them. By incorporating four scores during inference, RecoBERT can infer text-based item-to-item similarities more accurately than other techniques. In addition, we introduce a new language understanding task for wine recommendations using similarities based on professional wine reviews. As an additional contribution, we publish annotated recommendations dataset crafted by human wine experts. Finally, we evaluate Re-coBERT and compare it to various state-of-theart NLP models on wine and fashion recommendations tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers4
- Interpreting BERT-based Text Similarity via Activation and Saliency MapsItzik Malkiel, Dvir Ginzburg, Oren Barkan, Avi Caciularu et al.WWW 2022 · 28 citations
- MTAdam: Automatic Balancing of Multiple Training Loss TermsItzik Malkiel, Lior WolfEMNLP 2021 · 12 citations
- Evaluating Short-Term Temporal Fluctuations of Social Biases in Social Media Data and Masked Language ModelsYi Zhou, Danushka Bollegala, José Camacho-ColladosEMNLP 2024 · 3 citations
- SignFlow Bipartite Subgraph Network For Large-Scale Graph Link Sign PredictionYixiao Zhou, Xiaoqing Lyu, Hongxiang Lin, Huiying Hu et al.NeurIPS 2025
Builds on4
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-TrainingHangbo Bao, Li Dong, Furu Wei, Wenhui Wang et al.ICML 2020 · 423 citations
- Scalable Attentive Sentence Pair Modeling via Distilled Sentence EmbeddingOren Barkan, Noam Razin, Itzik Malkiel, Ori Katz et al.AAAI 2020 · 37 citations
- Improving Word Embedding Factorization for Compression using Distilled Nonlinear Neural DecompositionVasileios Lioutas, Ahmad Rashid, Krtin Kumar, Md. Akmal Haidar et al.EMNLP Findings 2020 · 8 citations
Related papers
- RecBase: Generative Foundation Model Pretraining for Zero-Shot RecommendationSashuai Zhou, Weinan Gan, Qijiong Liu, Ke Lei et al.EMNLP 2025 · 1 citation
- U-BERT: Pre-training User Representations for Improved RecommendationZhaopeng Qiu, Xian Wu, Jingyue Gao, Wei FanAAAI 2021 · 171 citations
- ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation TransferYuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang et al.ACL 2021
- FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and CaptioningSuvir Mirchandani, Licheng Yu, Mengjiao Wang, Animesh Sinha et al.EMNLP 2022 · 9 citations
- Enhancing Large Language Model Based Sequential Recommender Systems with Pseudo Labels ReconstructionHyunsoo Na, Minseok Gang, Youngrok Ko, Jinseok Seol et al.EMNLP Findings 2024 · 3 citations
