ATX: Accelerator Task Extensions

Integrating accelerators in CPU multicores combines the benefits of accelerated computation with the flexibility and programmability of CPUs. CPU-integrated accelerators can be classified into In-Core Accelerators (ICAs), which typically reside inside the core's pipeline, and Out-of-Core Accelerators (OCAs), which are typically attached at the core's cache subsystem. Both designs have shortcomings: ICAs can be bottlenecked by the core's general-purpose memory access interface, while OCAs' core-accelerator interface can limit execution overlap and expose communication overheads. In this paper we introduce Near-Core Accelerators (NCAs), a new class of accelerators that aims to combine the advantages of ICAs and OCAs, and address their shortcomings. Specifically, NCAs have their own read interface to the memory system, eliminating ICAs' bottleneck. At the same time, NCAs can be invoked speculatively and out-of-order, enabling both coreaccelerator execution overlap and low-overhead core-accelerator communication. We also propose the Accelerator Task Extensions (ATX), a set of instructions and hardware extensions to support NCAs. With ATX instructions, CPU cores can speculatively invoke a diverse range of NCAs. ATX includes the Unified Transfer Engine (UTE), a programmable hardware module that efficiently supplies data to NCAs and virtualizes the interface between the CPU core and the NCAs. The UTE interfaces the CPU core with multiple NCAs and the cache subsystem, fetching and prefetching accelerator data, and scheduling tasks to potentially multiple NCAs transparently to the CPU core. We evaluate ATX NCAs with a variety of important kernels from machine learning and scientific computing. We show that ATX NCAs accelerate these kernels by $\mathbf{1. 3 - 1 8} \times$ over various CPUintegrated accelerator alternatives.