From Memorization to Generalization: A Practical Neural Network Prefetching Framework

Data prefetchers are instrumental in mitigating the memory wall by anticipating future memory accesses. The firstlevel (L1) cache is the ideal location for prefetching, as it observes the complete, unfiltered stream of memory requests. However, its severely limited hardware resources have historically restricted L1 prefetchers to simple, pattern-based strategies, which struggle with complex access patterns. Conversely, recent machine learning (ML) based prefetchers like Pythia and Pathfinder demonstrate broader coverage but their prohibitive storage and computational costs relegate them to the L2 or last-level cache, making them too slow to be effective for the L1. This creates a critical design tension: the most powerful prefetching intelligence is stranded far from where it is needed most. To resolve this tension, we propose Moirai, a practical neural network prefetching framework designed specifically for the L1 data cache. Moirai's core is CaPNet, a highly compact Binarized Neural Network we designed to achieve high-accuracy predictions within a tiny hardware footprint. Our evaluation shows that Moirai delivers competitive performance against state-of-theart prefetchers while consuming only 780 Bytes of storage, an area reduction of over 97% compared to recent ML-based designs. Moirai thus charts a practical path forward for deploying powerful, generalization-based predictors in the most resourcesensitive parts of a modern processor.