TarGATE: Target-Aware Data Selection via Token-Attenuation Gates

Targeted instruction tuning requires selecting pertinent samples from massive mixed candidate datasets guided by a small reference dataset reflecting the desired capability. However, efficiently identifying high-quality data amidst noise remains challenging. To address this, we propose Target-aware GATEs (TarGATE), a simple yet effective data selection framework that leverages the model's inherent data understanding ability. These gates compute a token-level Information Retention Ratio (IRR) to attenuate the output of the feed-forward network, where the instancelevel average IRR serves as a quantitative metric for data quality. To align gates' preferences with the target task, we employ a joint optimization strategy utilizing the reference dataset and a subset of candidate data, which encourages the gates to assign higher IRRs to reference-aligned data while suppressing low-quality samples. Extensive experiments across noisy and real-world scenarios demonstrate that TarGATE outperforms related baselines. Furthermore, TarGATE exhibits superior computational efficiency and strong crossmodel transferability, enabling smaller selector to effectively curate high-quality fine-tuning data for larger foundation models. The code is available here.