Synergy-of-Experts: Collaborate to Improve Adversarial Robustness

Learning adversarially robust models requires invariant predictions to a small neighborhood of its natural inputs, often encountering insufficient model capacity. There is research showing that learning multiple sub-models in an ensemble could mitigate this insufficiency, further improving the generalization and the robustness. However, the ensemble's voting-based strategy excludes the possibility that the true predictions remain with the minority. Therefore, this paper further improves the ensemble through a collaboration scheme-Synergy-of-Experts (SoE). Compared with the voting-based strategy, the SoE enables the possibility of correct predictions even if there exists a single correct sub-model. In SoE, every sub-model fits its specific vulnerability area and reserves the rest of the sub-models to fit other vulnerability areas, which effectively optimizes the utilization of the model capacity. Empirical experiments verify that SoE outperforms various ensemble methods against white-box and transfer-based adversarial attacks. The source codes are available at https://github.com/cuis15/synergy-of-experts . * The first two authors have made equal contributions. Definition 1 (best-performing sub-model). Given an input x and its label y, the best-performing sub-model achieves the lowest objective loss on the data (x, y) among all M sub-models: Note that the best-performing sub-model is w.r.t. the input data (x, y), i.e., different input data correspond to different best-performing sub-models. Collaboration. Suppose that there are M learned sub-models f θ1 , f θ2 , ..., f θ M . Given an input x, sub-models make predictions f θ1 (x), f θ2 (x), ..., f θ M (x) . The collaboration tries to output a final prediction collaboration(x, f θ1 , ..., f θ M ) by the best-performing sub-model: collaboration(x, f θ1 , ..., f θ M ) = f θ best (x).