RaPiD: AI Accelerator for Ultra-low Precision Training and Inference

The growing prevalence and computational demands of Artificial Intelligence (AI) workloads has led to widespread use of hardware accelerators in their execution. Scaling the performance of AI accelerators across generations is pivotal to their success in commercial deployments. The intrinsic error-resilient nature of AI workloads present a unique opportunity for performance/energy improvement through precision scaling. Motivated by the recent algorithmic advances in precision scaling for inference and training, we designed RAPID 1 , a 4-core AI accelerator chip supporting a spectrum of precisions, namely, 16 and 8-bit floating-point and 4 and 2-bit fixed-point. The 36mm 2 RAPID chip fabricated in 7nm EUV technology delivers a peak 3.5 TFLOPS/W in HFP8 mode and 16.5 TOPS/W in INT4 mode at nominal voltage. Using a performance model calibrated to within 1% of the measurement results, we evaluated DNN inference using 4-bit fixed-point representation for a 4-core 1 RAPID chip system and DNN training using 8-bit floating point representation for a 768 TFLOPs AI system comprising 4 32-core RAPID chips. Our results show INT4 inference for batch size of 1 achieves 3 -13.5 (average 7) TOPS/W and FP8 training for a mini-batch of 512 achieves a sustained 102 -588 (average 203) TFLOPS across a wide range of applications.