CoLES: Contrastive Learning for Event Sequences with Self-Supervision
Dmitrii Babaev, Nikita Ovsov, Ivan Kireev, Mariya Ivanova, Gleb Gusev, Ivan Nazarov, Alexander Tuzhilin
Abstract
We address the problem of self-supervised learning on discrete event sequences generated by real-world users. Self-supervised learning incorporates complex information from the raw data in low-dimensional fixed-length vector representations that could be easily applied in various downstream machine learning tasks. In this paper, we propose a new method "CoLES", which adapts contrastive learning, previously used for audio and computer vision domains, to the discrete event sequences domain in a self-supervised setting.
We deployed CoLES embeddings based on sequences of transactions at the large European financial services company. Usage of CoLES embeddings significantly improves the performance of the pre-existing models on downstream tasks and produces significant financial gains, measured in hundreds of millions of dollars yearly. We also evaluated CoLES on several public event sequences datasets and showed that CoLES representations consistently outperform other methods on different downstream tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- A Simple Framework for Contrastive Learning of Visual Representations cites Momentum Contrast for Unsupervised Visual Representation Learning
- Momentum Contrast for Unsupervised Visual Representation Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Disentangled Self-Supervision in Sequential Recommenders cites Momentum Contrast for Unsupervised Visual Representation Learning
- GradGCL: Gradient Graph Contrastive Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- LightPath: Lightweight and Scalable Path Representation Learning cites Momentum Contrast for Unsupervised Visual Representation Learning
Cited by top-tier papers2
Builds on5
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than GeneratorsKevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. ManningICLR 2020 · 541 citations
- Disentangled Self-Supervision in Sequential RecommendersJianxin Ma, Chang Zhou, Hongxia Yang, Peng Cui et al.KDD 2020 · 223 citations
- Momentum Contrast for Unsupervised Visual Representation LearningKaiming He, Haoqi Fan, Yuxin Wu, Saining Xie et al.CVPR 2020
Related papers
- SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized Sequence RepresentationsHooman Sedghamiz, Shivam Raval, Enrico Santus, Tuka Alhanai et al.EMNLP Findings 2021 · 19 citations
- REBAR: Retrieval-Based Reconstruction for Time-series Contrastive LearningMaxwell A. Xu, Alexander Moreno, Hui Wei, Benjamin M. Marlin et al.ICLR 2024 · 23 citations
- Contrastive Learning for Sequential RecommendationXu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu et al.ICDE 2022 · 674 citations
- Disentangling Long and Short-Term Interests for RecommendationYu Zheng, Chen Gao, Jianxin Chang, Yanan Niu et al.WWW 2022 · 128 citations
- Intent Contrastive Learning for Sequential RecommendationYongjun Chen, Zhiwei Liu, Jia Li, Julian J. McAuley et al.WWW 2022 · 429 citations
