Lune

NeurIPS2025Top-tier venue

Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs

Masahiro Kaneko, Timothy Baldwin

Abstract

Adversarial attacks by malicious users that threaten the safety of large language models (LLMs) can be viewed as attempts to infer a target property T that is unknown when an instruction is issued, and becomes knowable only after the model's reply is observed. Examples of target properties T include the binary flag that triggers an LLM's harmful response or rejection, and the degree to which information deleted by unlearning can be restored, both elicited via adversarial instructions. The LLM reveals an observable signal Z that potentially leaks hints for attacking through a response containing answer tokens, thinking process tokens, or logits. Yet the scale of information leaked remains anecdotal, leaving auditors without principled guidance and defenders blind to the transparency-risk trade-off. We fill this gap with an information-theoretic framework that computes how much information can be safely disclosed, and enables auditors to gauge how close their methods come to the fundamental limit. Treating the mutual information I(Z; T ) between the observation Z and the target property T as the leaked bits per query, we show that achieving error ε requires at least log(1/ε)/I(Z; T ) queries, scaling linearly with the inverse leak rate and only logarithmically with the desired accuracy. Thus, even a modest increase in disclosure collapses the attack cost from quadratic to logarithmic in terms of the desired accuracy. Experiments on seven LLMs across system-prompt leakage, jailbreak, and relearning attacks corroborate the theory: exposing answer tokens alone requires about a thousand queries; adding logits cuts this to about a hundred; and revealing the full thinking process trims it to a few dozen. Our results provide the first principled yardstick for balancing transparency and security when deploying LLMs.

Recent work has introduced a variety of adaptive attacks, from gradient-guided prompt search and CoT-based editing to self-play strategies [56,48,51]. However, evaluation remains overwhelmingly

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Language Models are Zero-Shot Reasoners cites Language Models are Few-Shot Learners
  • Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attacks cites Language Models are Few-Shot Learners
  • Language Models are Few-Shot Learners cites The Curious Case of Neural Text Degeneration
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak AttacksMasahiro Kaneko, Ayana Niwa, Timothy BaldwinICLR 2026 · 6 citations

Builds on18

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  4. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  5. WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language ModelsLiwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger et al.NeurIPS 2024 · 247 citations

Related papers

  1. Resolving the Security-Auditability Dilemma with Auditable Latent Chain-of-Thought AlignmentGuan Wang, Biyu Zhou, Xuehai Tang, Jizhong Han et al.ACL 2026
  2. Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI ResponsesDavid Glukhov, Ziwen Han, Ilia Shumailov, Vardan Papyan et al.ICLR 2025
  3. Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?Yuan Xin, Dingfan Chen, Linyi Yang, Michael Backes et al.ACL Findings 2026
  4. Protecting Your LLMs with Information BottleneckZichuan Liu, Zefan Wang, Linjie Xu, Jinyu Wang et al.NeurIPS 2024 · 43 citations
  5. Adversarial Reasoning at Jailbreaking TimeMahdi Sabbaghi, Paul Kassianik, George J. Pappas, Amin Karbasi et al.ICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines