Lune

EMNLP Findings2025

MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages

Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl

摘要

Subword-based tokenization methods often fail to preserve morphological boundaries, a limitation especially pronounced in low-resource, morphologically complex languages such as those written in the Ge'ez script. To address this, we present Morpheme-aware Subword Vocabulary Construction (MoVoC) and train MoVoC-Tok, a tokenizer that integrates supervised morphological analysis into the subword vocabulary. This hybrid segmentation approach combines morpheme-based and Byte Pair Encoding (BPE) tokens to preserve morphological integrity while maintaining lexical meaning. To tackle resource scarcity, we release morpheme-annotated data for four Ge'ez script languages (169,806 records) and a morphemeaware vocabulary for two of them. Scoring every comparison on words held out from the morpheme resource that constrains the MoVoC-Tok applied to that language, MoVoC-Tok has the highest MorphScore in three of four languages and the highest morpheme boundary precision in two of four, with its largest and most clearly resolved gains on Tigre, a language it was not trained on and to which it is applied cross-lingually. The machine translation results reported in the published version are withdrawn. Data, tokenizers and evaluation code are publicly available. 1

This is a corrected version of Teklehaymanot et al. (2025). The evaluation artifacts (checkpoints, predictions, scoring pipeline) behind the results in the published version were not recovered, and the published figures could not be reproduced from the released code and data. Following the advice of the EMNLP 2025 programme chairs, every result in this version is replaced by results reconstructed from the public repository. The method (Section 3) is unchanged. The main corrections are:

1 https://github.com/hailaykidu/MoVoC

• Examples (Tables 1 and 6). The released tokenizers do not produce the segmentations shown in the published Table 1 and Section 7, and the published translation example could not be reproduced. Both are replaced by actual tokenizer output.

• Setup. Training hyperparameters, evaluation data and vocabulary sizes are corrected to those recorded in the repository.

问 Lune

尽管问,回答会引用原文。

Lune 会读完全文和相关的 Paper,每个回答都附上所依据的原文段落。

可以从这些问题问起

Lune 的依据

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

它引用的 Paper1

  1. Lexically Grounded Subword SegmentationJindrich Libovický, Jindrich HelclEMNLP 2024 · 2 次引用

相关 Paper

  1. Multilingual Tokenization through the Lens of Indian Languages: Challenges and InsightsMaharaj Brahma, Karthika NJ, Rajat Verma, Nagasai Saketh Naidu 等ACL Findings 2026
  2. Length-aware Byte Pair Encoding for Mitigating Over-segmentation in Korean Machine TranslationJungseob Lee, Hyeonseok Moon, Seungjun Lee, Chanjun Park 等ACL Findings 2024 · 1 次引用
  3. KinyaBERT: a Morphology-aware Kinyarwanda Language ModelAntoine Nzeyimana, Andre Niyongabo RubungoACL 2022 · 45 次引用
  4. Exploring morphology-aware tokenization: A case study on Spanish language modelingAlba Táboas García, Piotr Przybyla, Leo WannerEMNLP 2025 · 1 次引用
  5. MorphBPE: Morphology-Aware Tokenization for Efficient LLM TrainingEhsaneddin Asgari, Yassine El Kheir, MohammadAli SadraeiJavaheri, Ali NazariACL Findings 2026
黄昏的海面,两侧是细线勾勒的悬崖