Instance-adaptive training with noise-robust losses against noisy labels

In order to alleviate the huge demand for annotated datasets for different tasks, many recent natural language processing datasets have adopted automated pipelines for fast-tracking usable data. However, model training with such datasets poses a challenge because popular optimization objectives are not robust to label noise induced in the annotation generation process. Several noise-robust losses have been proposed and evaluated on tasks in computer vision, but they generally use a single dataset-wise hyperparamter to control the strength of noise resistance. This work proposes novel instance-adaptive training frameworks to change dataset-wise hyperparameters of noise resistance in such losses to be instance-specific. Such instance-specific noise resistance hyperparameters are predicted by special instance-level label quality predictors, which are trained along with the main models. Experiments on noisy and corrupted NLP datasets show that proposed instance-adaptive training frameworks help increase the noiserobustness provided by such losses, promoting the use of the frameworks and associated losses in training NLP models with noisy data.