FedDebug: Systematic Debugging for Federated Learning Applications

In Federated Learning (FL), clients independently train local models and share them with a central aggregator to build a global model. Impermissibility to access clients' data and collaborative training make FL appealing for applications with data-privacy concerns, such as medical imaging. However, these FL characteristics pose unprecedented challenges for debugging. When a global model's performance deteriorates, identifying the responsible rounds and clients is a major pain point. Developers resort to trial-and-error debugging with subsets of clients, hoping to increase the global model's accuracy or let future FL rounds retune the model, which are time-consuming and costly. We design a systematic fault localization framework, FEDDE-BUG, that advances the FL debugging on two novel fronts. First, FEDDEBUG enables interactive debugging of realtime collaborative training in FL by leveraging record and replay techniques to construct a simulation that mirrors live FL. FEDDEBUG's breakpoint can help inspect an FL state (round, client, and global model) and move between rounds and clients' models seamlessly, enabling a fine-grained step-by-step inspection. Second, FEDDEBUG automatically identifies the client(s) responsible for lowering the global model's performance without any testing data and labels-both are essential for existing debugging techniques. FEDDEBUG's strengths come from adapting differential testing in conjunction with neuron activations to determine the client(s) deviating from normal behavior. FEDDEBUG achieves 100% accuracy in finding a single faulty client and 90.3% accuracy in finding multiple faulty clients. FEDDEBUG's interactive debugging incurs 1.2% overhead during training, while it localizes a faulty client in only 2.1% of a round's training time. With FEDDEBUG, we bring effective debugging practices to federated learning, improving the quality and productivity of FL application developers.