Adaptive Data and Task Joint Scheduling for Multi-Task Learning

Multi-task Learning (MTL) involves training multiple tasks within a single model to improve overall performance by leveraging shared knowledge. However, this joint training can result in performance degradation due to task conflicts, typically manifesting as conflicts in task gradients. Existing solutions primarily focus on modeling task gradient relationships, which overlook the differences in how the same data sample influences different tasks. These differences are the source of intricate task gradient relationships and could further lead to varying degrees of impact from conflicts on tasks. To tackle these challenges, we propose DTJS, a novel adaptive Data and Task Joint Scheduling approach for MTL, which uniquely considers the influence of data within each task and the distinct task perception of gradient conflicts from an innovative scheduling perspective. Specifically, we design intra-task scheduling to quantify the difficulty level of the data based on its influence within each task, facilitating easy-to-hard data scheduling. Concurrently, inter-task scheduling is proposed to capture the diverse relationship among joint learning tasks via assessing the severity of conflicts between tasks and adaptively considering their effects on individual tasks through learnable task conflict perception. Furthermore, DTJS utilizes a bi-level optimization strategy that alternately updates model parameters and the learnable task conflict perception, taking into account their interdependence. Scheduled model gradients are used to optimize the MTL model, while implicit gradients refine the learnable task conflict perception. Extensive experimental results not only demonstrate that DTJS improves the performance of the MTL model over SOTA methods across various scenarios but also explain how DTJS schedules both data and tasks to bring performance improvements. The code is available at https://github.com/ZeyuLiu0706IDTJS.