Lune

CVPR Findings2026

Safe Codebook: Token-Level Moderation for Safer Visual Autoregressive Generation

Jiaxuan Zhang, Qianqian Xu, Peisong Wen, Siran Dai, Yang Liu, Qingming Huang

Abstract

Recently, Visual Autoregressive (VAR) models have emerged for text-to-image (T2I) generation, offering competitive image quality and faster generation speed, yet safety mechanisms for VAR remain underexplored. Although guidancebased methods such as Safe Latent Diffusion (SLD) can be transferred to VAR models, the resulting variant (SLVAR) suffers from two limitations: 1) a single guidance embedding recalls harmful content incompletely, leading to weak suppression, and 2) the modified features are not constrained to remain close to the original image, reducing content fidelity. To address these issues, we propose a training-free framework named Safe Codebook (SaCo), which leverages the VAR codebook to improve safety. Specifically, SaCo performs cross-modal dictionary lookup to construct a harmful book for more complete recall of harmful tokens, and adopts a token-level replacement scheme that replaces only harmful tokens with benign ones well matched to the original image. Extensive experiments show that SaCo achieves stronger harmful content suppression and higher original content fidelity than diffusion-based safety baselines, without modifying model weights. Code is available at https://github.com/ jiaxuanzhang1/SaCo.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Zero-Shot Text-to-Image Generation
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
  • Scalable Diffusion Models with Transformers cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Scalable Diffusion Models with Transformers cites Zero-Shot Text-to-Image Generation
  • Scalable Diffusion Models with Transformers cites Denoising Diffusion Probabilistic Models
  • Directly Denoising Diffusion Models cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Directly Denoising Diffusion Models cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on34

  1. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  2. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  3. Directly Denoising Diffusion ModelsDan Zhang, Jingjing Wang, Feng LuoICML 2024 · 11,724 citations
  4. Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations
  5. Scalable Diffusion Models with TransformersWilliam Peebles, Saining XieICCV 2023 · 5,568 citations

Related papers

  1. Closing the Safety Gap: Surgical Concept Erasure in Visual Autoregressive ModelsXinhao Zhong, Yimin Zhou, Zhiqi Zhang, Junhao Li et al.ICLR 2026 · 7 citations
  2. Training-Free Safe Text Embedding Guidance for Text-to-Image Diffusion ModelsByeonghu Na, Mina Kang, Jiseok Kwak, Minsang Park et al.NeurIPS 2025 · 8 citations
  3. Safe Autoregressive Image Generation with Iterative Self-Improving CodebooksYunqi Xue, Zhijiang Li, Phil Torr, Jindong GuICML 2026 · 1 citation
  4. Training-Free Text-Guided Image Editing with Visual Autoregressive ModelYufei Wang, Lanqing Guo, Zhihao Li, Jiaxing Huang et al.ICCV 2025
  5. The Safety-Aware Denoiser for Text Diffusion ModelsAmman Yusuf, Zhejun Jiang, Mi Jung ParkICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines