Responsible Text-to-Image Diffusion: Interpretable and Linearly Controllable Semantics for Fair and Safe Generation

Text-to-image (T2I) diffusion models (DMs) have achieved remarkable generative quality but still exhibit risk of producing biased and inappropriate images. A promising line of prior work aims to mitigate this issue by learning interpretable, linearly controllable concepts from semantic spaces, such as the U-Net bottleneck; however, these methods rely entirely on U-Net architectures and cannot be generalized to modern ViT-based DMs, including FLUX and PixArt. In this work, we present an architecture-agnostic framework for discovering interpretable and linearly controllable semantic attributes across any T2I DM backbone. Theoretically, we show that multi-modal attention heads in ViT-based DMs exhibit a linear semantic structure: injected concept vectors combine linearly at the attention level and induce near-linear effects at the model output, satisfying homogeneity and additivity. These theoretical results are aligned with and supported by empirical experiments. Building on this insight, we introduce a method that learns external concept vectors, which are added to the multi-modal attention heads for ViT-based DMs or to the bottleneck layer for U-Net-based DMs, while keeping pretrained models frozen. Experiments across SDXL, SD3.5, PixArt, and FLUX demonstrate that these concept vectors provide interpretability, linearity, and significantly improved fairness while preserving visual fidelity. The code and demo are available at https://github.com/Moslem-Sh21/responsible-t2i-diffusion.