RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks

Incremental learning aims to adapt to new sets of categories over time with minimal computational overhead. Prior work often trains efficient task-specific adapters that modify frozen layer weights or features to capture relevant information without affecting predictions on any previously learned categories. While these adapters are generally more efficient than finetuning the entire network, they still can require tens or hundreds of thousands of task-specific trainable parameters, even for relatively small networks. This is can be problematic for resourceconstrained environments with high communication costs, such as edge devices or mobile phones. Thus, we propose Reparameterized, Compact weight Adaptation for Sequential Tasks (RECAST), a novel method that dramatically reduces the number of task-specific trainable parameters to fewer than 50 -several orders of magnitude less than competing methods like LoRA. RECAST accomplishes this efficiency by learning to decompose layer weights into a soft parameter-sharing framework consisting of a set of shared weight templates and very few modulespecific scaling factor coefficients. This soft parameter-sharing framework allows for effective task-wise reparameterization by tuning only these coefficients while keeping the templates frozen. A key innovation of RECAST is the novel weight reconstruction pipeline called Neural Mimicry, which eliminates the need for training in our framework from scratch. Extensive experiments across six diverse datasets demonstrate RECAST outperforms the state-of-the-art by up to ∼ 1.5% and improves baselines > 3% across various scales, architectures, and parameter spaces. Moreover, we show that RECAST's architecture-agnostic nature allows for seamless integration with existing methods, further boosting performance 1 .