BanglaByT5: Byte-Level Modelling for Bangla

Large language models (LLMs) have achieved remarkable success across various natural language processing tasks. However, most LLM models use traditional tokenizers like BPE and SentencePiece, which fail to capture the finer nuances of a morphologically rich language like Bangla (Bengali). In this work, we introduce BanglaByT5, the first bytelevel encoder-decoder model explicitly tailored for Bangla. Built upon a small variant of Google's ByT5 architecture, BanglaByT5 is pre-trained on a 14GB curated corpus combining high-quality literary and newspaper articles. Through zero-shot and supervised evaluations across generative and classification tasks, BanglaByT5 demonstrates competitive performance, surpassing several multilingual and larger models. Our findings highlight the potential of BanglaByT5 as a lightweight yet powerful tool for Bangla NLP, particularly in resource-constrained and scalable environments. BanglaByT5 is publicly available for download from https://huggingface.co/ Vacaspati/BanglaByT5.