DA-DFGAS: Differentiable Federated Graph Neural Architecture Search with Distribution-Aware Attentive Aggregation

To preserve user privacy while enabling mobile intelligence, techniques have been proposed to train deep neural networks on decentralized data. However, decentralized training makes the design of neural architecture quite difficult as it already was. Such difficulty is further amplified when designing and deploying different neural architectures for heterogeneous mobile platforms. In this work, we propose an automatic neural architecture search into the decentralized training, as a new DNN training paradigm called Federated Neural Architecture Search, namely federated NAS. To deal with the primary challenge of limited on-client computational and communication resources, we present DecNAS, a highly optimized framework for efficient federated NAS. DecNAS fully exploits the key opportunity of insufficient model candidate re-training during the architecture search process, and incorporates three key optimizations: parallel candidates training on partial clients, early dropping candidates with inferior performance, and dynamic round numbers. Tested on large-scale datasets and typical CNN architectures, DecNAS achieves comparable model accuracy as state-of-theart NAS algorithm that trains models with centralized data, and also reduces the client cost by up to 200× or more compared to a straightforward design of federated NAS. Model Search Model Training Data feedback Model Search Data Model Training Model Fusion (a) Traditional NAS (b) Federated NAS