Lune

NeurIPS2024Top-tier venue

Statistical and Geometrical properties of the Kernel Kullback-Leibler divergence

Anna Korba, Francis R. Bach, Clémentine Chazal

Abstract

In this paper, we study the statistical and geometrical properties of the Kullback-Leibler divergence with kernel covariance operators (KKL) introduced by Bach [2022]. Unlike the classical Kullback-Leibler (KL) divergence that involves density ratios, the KKL compares probability distributions through covariance operators (embeddings) in a reproducible kernel Hilbert space (RKHS), and compute the Kullback-Leibler quantum divergence. This novel divergence hence shares parallel but different aspects with both the standard Kullback-Leibler between probability distributions and kernel embeddings metrics such as the maximum mean discrepancy. A limitation faced with the original KKL divergence is its inability to be defined for distributions with disjoint supports. To solve this problem, we propose in this paper a regularised variant that guarantees that the divergence is well defined for all distributions. We derive bounds that quantify the deviation of the regularised KKL to the original one, as well as finite-sample bounds. In addition, we provide a closed-form expression for the regularised KKL, specifically applicable when the distributions consist of finite sets of points, which makes it implementable. Furthermore, we derive a Wasserstein gradient descent scheme of the KKL divergence in the case of discrete distributions, and study empirically its properties to transport a set of points to a target distribution.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Kernel Stein Discrepancy Descent cites SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence
  • Score-Based Generative Modeling through Stochastic Differential Equations cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Flowing Datasets with Wasserstein over Wasserstein Gradient FlowsClément Bonet, Christophe Vauthier, Anna KorbaICML 2025
  2. Sinkhorn Normalization of Diffusion KernelsNathan Kessler, Robin Magnet, Jean FeydyICML 2026
  3. A Principled Path to Fitted Distributional EvaluationSungee Hong, Jiayi Wang, Zhengling Qi, Raymond K. W. WongNeurIPS 2025

Builds on11

  1. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  2. Score-Based Generative Modeling through Stochastic Differential EquationsYang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar et al.ICLR 2021 · 1,270 citations
  3. SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergenceSinho Chewi, Thibaut Le Gouic, Chen Lu, Tyler Maunu et al.NeurIPS 2020 · 92 citations
  4. Kernel Stein Discrepancy DescentAnna Korba, Pierre-Cyril Aubin-Frankowski, Szymon Majewski, Pierre AblinICML 2021 · 64 citations
  5. KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint SupportPierre Glaser, Michael Arbel, Arthur GrettonNeurIPS 2021 · 49 citations

Related papers

  1. Well-Posed KL-Regularized Control via Wasserstein and Kalman–Wasserstein KL DivergencesViktor Stein, Adwait Datar, Nihat AyICML 2026
  2. Kernel Quantile Embeddings and Associated Probability MetricsMasha Naslidnyk, Siu Lun Chau, François-Xavier Briol, Krikamol MuandetICML 2025
  3. Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert SpaceSandesh Ghimire, Aria Masoomi, Jennifer G. DyNeurIPS 2021 · 17 citations
  4. The Unbalanced Gromov Wasserstein Distance: Conic Formulation and RelaxationThibault Séjourné, François-Xavier Vialard, Gabriel PeyréNeurIPS 2021 · 106 citations
  5. A Non-Asymptotic Analysis for Stein Variational Gradient DescentAnna Korba, Adil Salim, Michael Arbel, Giulia Luise et al.NeurIPS 2020 · 102 citations
Dusk over the sea between two cliffs drawn in fine vertical lines