HiPACK: Efficient Sub-8-Bit Direct Convolution with SIMD and Bitwise Management

Quantized Deep Neural Networks (DNNs) have progressed to utilize sub-8-bit data types, achieving notable reductions in both memory usage and computational expenses.Nevertheless, the efficient execution of sub-8-bit convolution operations remains insufficiently optimized, especially in the context of Single Instruction Multiple Data (SIMD) architectures.This paper introduces HiPACK, which takes the packing for efficient convolution as foundation to reduce the required number of operations, and address the challenges of SIMD incompatibility in sub-8-bit direct convolution by decoupling the unpacking phase from multiplication operations, optimizing the caching of intermediate data, determining the ideal segmentation, and employing a dual interleaved register (DIR) mechanism to facilitate parallel multiplication through SIMD.Collectively, these optimizations significantly enhance computational efficiency and diminish operational overhead.Evaluation on the BCM2711 ARM processor reveals speedups of up to 4.6× compared to floating-point calculations and 1.7× when measured against state-of-the-art solutions, thus accelerating model inference.Our implementation is open-sourced and available at: https://github.com/Xtra-Computing/HIPACK.