LUCID: An Updatable and Concurrent Learned Index for Larger-Than-Memory Data Management

Learned indexes have shown great potential for improving data storage and access performance, but their adoption in real-world systems is limited when data exceed memory capacity. In-memory approaches become impractical at scale, while fully on-disk techniques often fail to meet the performance demands of diverse workloads. Hybrid storage architectures that combine memory's low latency with disk's high capacity offer a promising middle ground. However, deploying learned indexes in such environments, especially under dynamic and complex workloads, remains challenging. An effective solution must manage larger-than-memory datasets within constrained resources, support dynamic updates, and enable concurrent operations-requirements that existing methods do not satisfy simultaneously. This paper presents LUCID, an Updatable and Concurrent Learned Index for larger-than-memory Data management. LUCID is a learned tree structure that indexes data spanning memory and disk, offering robust support for dynamic and concurrent workloads. First, it employs model-guided buffering to optimize in-memory insertions. Second, it introduces an overflow strategy at the leaf level to reduce the cost of structural modifications during updates. Third, it adopts a flattened design to track on-disk data more precisely while minimizing memory overhead. Finally, LUCID incorporates a lightweight, optimistic locking mechanism tailored to the needs of larger-than-memory systems, enabling efficient concurrent access.