SWIFT: A Scalable Lightweight Infrastructure for Fine-Tuning

Recent development in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have leveraged Attentionbased Transformer architectures and achieved superior performance and generalization capabilities. They have since covered extensive areas of traditional learning tasks. For instance, text-based tasks such as text classification and sequence labeling, as well as multi-modal tasks like Visual Question Answering (VQA) and Optical Character Recognition (OCR), which were previously addressed using different models, can now be tackled based on one foundation model. Consequently, the training and lightweight fine-tuning of LLMs and MLLMs, especially those based on the Transformer architecture, have become particularly important. In recognition of these overwhelming needs, we develop SWIFT, a customizable one-stop infrastructure for large models. With support of more than 550+ LLMs, 200+ MLLMs, and almost 200 Megatron models, SWIFT stands as the open-source framework that provides the most comprehensive support for fine-tuning large models. In particular, it is the first training framework that provides systematic support for MLLMs. In addition to the core functionalities of fine-tuning, SWIFT also integrates post-training processes such as inference, evaluation, and model quantization, to facilitate fast adoption of large models in various application scenarios. With a systematic integration of various training techniques, SWIFT offers helpful utilities, such as benchmark comparisons among different training techniques for large models. For fine-tuning models specialized in agent framework, we show that notable improvements on the Tool-Bench leaderboard can be achieved by training with customized datasets on SWIFT, with an increase of 5.2%-21.8% in the Act.EM metric over various baseline models, a reduction in hallucination by 1.6%-14.1%, and an average performance improvement of 8%-17%.