Minimal Impact ControlNet: Advancing Multi-ControlNet Integration

In current ControlNet training, each control is designed to influence all areas of an image, which can lead to conflicts when different control signals are expected to manage different parts of the image. Solution & Case • Introduce silent control signals: First introduce silent control signals that should remain inactive when other control signals are engaged, improving the compactness of the generation. • Feature injection and combination: Employ strategies based on multiobjective optimization principles to improve model performance. • Theoretical contribution: Develop and integrate a conservativity loss function within a large modular network architecture. Published as a conference paper at ICLR 2025 To tackle these challenges, adhering to the principle of "less is more", we introduce the Minimal Impact ControlNet (MIControlNet), a novel framework designed to refine the integration of multiple control signals within diffusion models. Our approach includes strategic modifications to the training data to reduce biases and utilizes a multi-objective optimization strategy during the feature combination phase, as well as addressing the asymmetry in the score function's Jacobian matrix induced by ControlNet. These methods aim to minimize conflicts between different control signals and between control signals and the inherent features of the dataset, thereby ensuring better compatibility and fidelity in the generated images. Canny Openpose ControlNet !.# MIControlNet(1-stage) ControlNet* ControlNet MIControlNet(2-stage)