gregfromstl / bantu-language-modeling Goto Github PK

This challenge involves language modeling for two closely-related languages in the Bantu language family: Swahili, spoken by more than 100 million people in eastern and southern Africa, and Kwere, spoken by about 100,000 people in central Tanzania. Bantu languages have complex morphology, especially the verbs; a single verbal stem can have thousands or even millions of different possible forms. For this reason, language technologies which require a fixed vocabulary of words are simply not feasible. One of the aims of this challenge is therefore to explore the possibility of building character-level language models for the Bantu languages. The other interesting research direction here stems from the fact that Swahili is relatively well-resourced in terms of NLP technology and corpora, while Kwere is not. There is a New Testament translation of Kwere available online, but very little else in terms of corpus material or social media presence in the language. But since the two languages are so similar linguistically and orthographically, this opens the possibility of “transferring” knowledge from Swahili to Kwere when learning a language model.