Combating the Memory Walls: Optimization Pathways for Long-Context Agentic Llm Inference

LLMs now form the backbone of AI agents for a diverse array of applications, including tool use, commandline interfaces, and web or computer interaction. These agentic LLM inference tasks are fundamentally different from chatbotfocused inference -they often have much larger context lengths to capture complex, prolonged inputs, such as an entire webpage DOM or complicated tool call trajectories. This, in turn, generates significant off-chip memory traffic for hardware at the inference stage and causes the workload to be constrained by the two memory walls, namely the bandwidth and capacity walls, preventing the compute units from achieving high utilization.

In this paper, we introduce PLENA, a hardware-software codesigned system that applies three core optimization pathways. PLENA features a novel flattened systolic-array architecture (Pathway 1) and efficient compute and memory units that support an asymmetric quantization scheme (Pathway 2). It also provides native support for FlashAttention (Pathway 3). In addition, PLENA is developed with a complete software-hardware stack, including a custom ISA, a compiler, a transaction-level simulator, and an automated design-space exploration flow. Experimental results show that PLENA delivers up to 2.23× and 4.70× higher throughput than the A100 GPU and TPU v6e, respectively, under identical multiplier counts and memory configurations during LLaMA agentic inference. PLENA also achieves up to 4.04× higher energy efficiency than A100 GPU. The full PLENA system-including its simulator, compiler, ISA, and RTL implementation-will be open-sourced to the research community.