Rare letters in toki pona [BUG] about common-voice HOT 4 OPEN

Hi @gtbot2007 and thanks so much for raising this issue, could I get a bit more detail in what's preventing uncommon letters from being used? Is this a limitation you've found in the guidelines that could be better clarified or are you getting a bug or error message when trying to input sentences that contain words like "yupekosi", "Pingo" and "kalamARR"?

from common-voice.

I believe at least the last one happens because of the default validation rules defined below, which tries to handle abbreviations:

For other cases you need to create language specific validation rules, like the ones here:
https://github.com/common-voice/common-voice/tree/main/server/src/core/sentences/validation/languages

from common-voice.

Since they are used so few words and the words are also ”non-standard” they are considered foreign letters. Which is technically true but maybe there should be a case to allow the words “Pingo”, “yupekosi”, “kalamARR”, “yutu” and maybe even “yu” and “y”.

from common-voice.

Sorry for the noise, I posted before checking. There IS actually a tok validation file here:
https://github.com/common-voice/common-voice/blob/main/server/src/core/sentences/validation/languages/tok.ts

It seems, each one of the samples you gave is hitting one regex rule there. Either they are not wanted, or the regex rules might need tweaks.

from common-voice.