ViLegalLM: Language Models for Vietnamese Legal Text

We present ViLegalLM, comprising ViLegal-BERT and ViLegalQwen, the first suite of Vietnamese pretrained language models for legal text understanding and generation. It includes one encoder-only model (ViLegal-BERT, 135M parameters) and two decoderonly models (ViLegalQwen2.5-1.5B-Base and ViLegalQwen3-1.7B-Base), all continually pretrained on a newly curated 16GB Vietnamese legal corpus, significantly larger than previous work. To mitigate data scarcity, we construct three synthetic datasets using LLM-based generation and hard negative mining for True/False QA, Multiple Choice QA, and Natural Language Inference. We establish state-of-theart results among open-source models on four main Vietnamese legal downstream tasks spanning ten benchmarks, demonstrating that continual pretraining from base models consistently outperforms instruction-tuned adaptation. Source codes, corpus, datasets, and model checkpoints are publicly available at https: //github.com/ntphuc149/ViLegalLM .