Perturbing to Preserve: Defending Fragile Knowledge in Online Continual Learning

Online continual learning mandates the ability to acquire knowledge from non-stationary data streams while preserving previously learned information, yet neural networks often forget. In this work, we uncover and systematically analyze a critical yet underexplored issue in this setting, which we term knowledge fragility: the phenomenon where correctly learned instances are abruptly forgotten following minor parameter updates. We attribute this phenomenon to two factors: (1) temporally, where high-frequency oscillations in parameter space lead to disproportionate forgetting relative to adaptation, and (2) spatially, where fragile instances reside in sharp, high-curvature regions of the loss landscape, making them highly susceptible to optimization noise. To counteract this fragility, we propose PDFK (Perturbing to Defend Fragile Knowledge)-a unified and task-agnostic framework that fortifies fragile knowledge along both temporal and spatial dimensions. Temporally, PDFK stabilizes longterm memory by employing exponential moving average (EMA) to suppress volatile parameter shifts. Spatially, it introduces lightweight, structured perturbations guided by consistency regularization, effectively flattening the local loss surface and enhancing robustness to future updates. Extensive experiments across diverse benchmarks demonstrate that PDFK consistently improves knowledge retention and surpasses state-of-the-art methods in both accuracy and forgetting metrics under challenging streaming settings.