NusaBERT Teaching IndoBERT to be Multilingual and Multicultural
Published in Proceedings of the Second Workshop in South East Asian Language Processing, 2025
Abstract
Indonesia’s linguistic landscape is remarkably diverse, encompassing over 700 languages and dialects, making it one of the world’s most linguistically rich nations. This diversity, coupled with the widespread practice of code-mixing and the presence of low-resource regional languages, presents unique challenges for modern pre-trained language models. In response to these challenges, we developed NusaBERT, building upon IndoBERT by incorporating vocabulary expansion and leveraging a diverse multilingual corpus that includes regional languages. Through rigorous evaluation across a range of benchmarks, NusaBERT demonstrates state-of-the-art performance in tasks involving multiple languages of Indonesia, paving the way for future natural language understanding research for under-represented languages.
BibTeX Citation
@inproceedings{wongso-etal-2025-nusabert,
title = {NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural},
author = {Wongso, Wilson and
Setiawan, David Samuel and
Limcorn, Steven and
Joyoadikusumo, Ananto},
editor = {Wijaya, Derry and
Aji, Alham Fikri and
Vania, Clara and
Winata, Genta Indra and
Purwarianti, Ayu},
booktitle = {Proceedings of the Second Workshop in South East Asian Language Processing},
month = jan,
year = {2025},
address = {Online},
publisher = {Association for Computational Linguistics},
url = {https://aclanthology.org/2025.sealp-1.2/},
pages = {10--26},
abstract = {We present NusaBERT, a multilingual model built on IndoBERT and tailored for Indonesia`s diverse languages. By expanding vocabulary and pre-training on a regional corpus, NusaBERT achieves state-of-the-art performance on Indonesian NLU benchmarks, enhancing IndoBERT`s multilingual capability. This study also addresses NusaBERT`s limitations and encourages further research on Indonesia`s underrepresented languages.}
}
Recommended citation: Wilson Wongso, David Samuel Setiawan, Steven Limcorn, and Ananto Joyoadikusumo. 2025. NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural. In Proceedings of the Second Workshop in South East Asian Language Processing, pages 10–26, Online. Association for Computational Linguistics.
Download Paper