Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updates

Deep Learning Recommendation Models (DLRMs) underpin personalized services but face a critical freshnessaccuracy tradeoff due to massive parameter synchronization overheads. Production DLRMs deploy decoupled training/inference clusters, where synchronizing petabyte-scale embedding tables (EMTs) causes multi-minute staleness, degrading recommendation quality and revenue. We observe that (1) inference nodes exhibit sustained CPU underutilization (peak$\leq 20 \%$), and (2) EMT gradients possess intrinsic low-rank structure, enabling compact update representation. We present LiveUpdate, a system that eliminates inter-cluster synchronization by colocating Low-Rank Adaptation (LoRA) trainers within inference nodes. LiveUpdate addresses two core challenges: (1) dynamic rank adaptation via singular value monitoring to constrain memory overhead ($<2 \%$of EMTs), and (2) NUMA-aware resource scheduling with hardware-enforced QoS to eliminate updateinference contention (P99 latency impact$<20 ~\text{ms}$). Evaluations show LiveUpdate reduces update costs by$2 \times$versus delta-update baselines while achieving higher accuracy within 1-hour windows. By transforming idle inference resources into freshness engines, LiveUpdate delivers online model updates while outperforming state- of-the-art delta-update methods by$\mathbf{0. 0 4 - 0. 2 4 \%}$in accuracy.