SpareTrain: Fault-Tolerant LLM Training via Low-Cost Dual Modular Redundancy

Dual Modular Redundancy (DMR) is a highly effective mechanism for detecting silent data corruption (SDC)—a critical reliability concern in large language model (LLM) training—by executing each operation twice. However, its high computation overhead has prevented practical deployment at scale. In this paper, we present SpareTrain, an LLM training system that achieves complete DMR with minimal overhead by repurposing the activation checkpointing mechanism and exploiting idle GPU time. Evaluations on up to 32 H200 GPUs show that SpareTrain improves throughput by 12–35% over naive DMR, corresponding to only 3–14% overhead compared to unprotected training, while maintaining full DMR error detection capabilities.