Hybrid Layout Control for Diffusion Transformer: Fewer Annotations, Superior Aesthetics

first fine-tunes the DiTs (e.g., SD3) to follow an anonymous layout, then continues fine-tuning the DiTs to follow the semantic layout, and finally includes a quality-tuning stage to enhance visual aesthetics. We show that this hybrid design is highly data-efficient, as we find only using a small amount of semantic layout annotations is sufficient, thereby significantly reducing dependency on regional prompts. In addition, we propose an efficient regional diffusion transformer to encode the spatial layout information using just a set of lower-resolution regional tokens instead of various carefully designed layout tokens. The region-wise diffusion loss over these regional tokens can guide the diffusion transformer learn to follow the given layout implicitly. We empirically validate the effectiveness of our approach by comparing it with the latest version of SiamLayout and show that our method achieves better results while being more than 10× more data efficient and ensuring superior aesthetics. Project Page: https://hybrid-layout-msra.github.io Attention Ratio: 80.86% Area Ratio: 67.72% Attention Ratio: 82.31% Area Ratio: 72.90% Attention Ratio: 79.40% Area Ratio: 67.19% Attention Ratio: 68.95% Area Ratio: 60.21% Attention Ratio: 81.47% Area Ratio: 76.22%