Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge

Recent advances in generative modeling have positioned diffusion models as stateof-the-art tools for sampling from complex data distributions. While these models have shown remarkable success across single-modality domains such as images and audio, extending their capabilities to Modality Translation (MT), translating information across different sensory modalities, remains an open challenge. Existing approaches often rely on restrictive assumptions, including shared dimensionality, Gaussian source priors, and modality-specific architectures, which limit their generality and theoretical grounding. In this work, we propose the Latent Denoising Diffusion Bridge Model (LDDBM), a general-purpose framework for modality translation based on a latent-variable extension of Denoising Diffusion Bridge Models. By operating in a shared latent space, our method learns a bridge between arbitrary modalities without requiring aligned dimensions. We introduce a contrastive alignment loss to enforce semantic consistency between paired samples and design a domain-agnostic encoder-decoder architecture tailored for noise prediction in latent space. Additionally, we propose a predictive loss to guide training toward accurate cross-domain translation and explore several training strategies to improve stability. Our approach supports arbitrary modality pairs and performs strongly on diverse MT tasks, including multi-view to 3D shape generation, image super-resolution, and multi-view scene synthesis. Comprehensive experiments and ablations validate the effectiveness of our framework, establishing a new strong baseline in general modality translation. For more information, see our project page: https://sites.google.com/view/lddbm/home.

structure or from low-resolution quality to a high-resolution one. Moreover, many implementations rely on neural architectures tailored to specific data types, such as U-Nets [52], which are well-suited to grid-based data but struggle with abstract or unstructured modalities. These limitations highlight the need for a general framework that minimizes reliance on modality-specific design and remains theoretically grounded across a broad range of cross-modal tasks.

Latent Diffusion Bridges such as [24,36,29,25,70] attempt to bridge the gap between diffusion models and general modality translation. However, most existing works adopt this formulation primarily for computational reasons-either to reduce the cost of the diffusion process itself [50,75], or to improve sampling efficiency through direct endpoint mappings [24] and cannot be applied to general MT purposes. In order to take a step towards a general effective framework, we suggest a new latent extension of DDBMs that facilitates generality for MT tasks along with effective performance.

LDDBM extends DDBMs to a shared latent space and learns a bridge between embeddings of disparate modalities, enabling translation without requiring the two domains to share dimensionality. Concretely (see Fig. 2), we: (i) encode source and target examples into a common latent space using simple modality-specific encoders; (ii) apply a latent diffusion bridge implemented with a Transformer denoiser that uses cross-attention to condition on the source latent while predicting the target latent; and (iii) decode the predicted latent back to the target modality. To promote semantic consistency, we utilize paired training data with a contrastive alignment loss (inspired by CLIP [48]) that pulls corresponding pairs together and pushes unrelated pairs apart. To ensure the bridge performs end-to-end translation, and not just local denoising, we add a predictive loss that compares the final decoded output to the ground-truth target. Finally, we reduce architectural bias with a domain-agnostic denoiser and use a simple yet effective iterative training scheme that alternates between alignment and denoising steps, improving stability and performance. This approach offers several key advantages. First, it supports arbitrary modality pairs without relying on fixed priors, shared dimensionality constraints, or hand-crafted architectures-aside from the encoder and decoder used to project data into the latent space. Second, the contrastive loss enhances semantic coherence across domains, and our bridge architecture achieves strong performance on a range of tasks, including multi-view to 3D shape generation [55], super-resolution [38], and scene generation from multi-view cameras [62]. Additionally, we provide a simple, flexible, and theoretically grounded framework for MT tasks and demonstrate its state-of-the-art performance. Finally, we conduct a thorough ablation study to evaluate each component of our framework, identifying the key elements that contribute to its improved performance.