Multi-orthography for Konkani - linking sentences collected in the gom and knn datasets about common-voice HOT 13 OPEN

https://discourse.mozilla.org/t/open-call-join-common-voice-in-introducing-variant-support-for-language-variants/128998

Checkout this post on MCV multi-orthography feature @alvynabranches @anniedempe

from common-voice.

Hi @chasingdragonflies

Thank you for the recommended adjustments and for the valuable insights you provided. While we're eager to implement the proposed modifications, it's important for us to consult with the community members and translators who requested the addition of the languages (knn and gom) to Common Voice, this is to ensure that we're accommodating the linguistic preferences of the entire community.

from common-voice.

Hello @thak123 @anniedhempe and @alvynabranches

As you already know, Konkani in देवनागरी (devanagari) is the official standard for reading and writing konkani in Goa. However there are variants or dialects of it spoken in smaller communities in the konkan region. Meaning, a person living in maharastra's konkan region will speak differently from person living in margao, and both of these people will speak differently than a person living in mangalore. Also, not all people will be able to read sentences in देवनागरी because mangalorean people read in kannada script. While many catholics in Goa prefer to read/speak in Romi-konkani.

There was a Mozilla Discourse post in July 2022 which talked about the introduction of Language Variants. Please watch the video linked in the post.

There is also another MCV post in March 2024 Multi-Orthography for language variants introducing support for languages with different writing system.

I am aware that currently the dataset is split into two locales for konkani. But, I think whole-heartedly that combining both of these datasets is possible because the difference of Romi and Devanagari is not much at all! The difference is mainly only in the pronunciation and writing script. With the upcoming introduction of multi-orthography for Language Variants that have multiple writing system/script, it is possible to use one locale (same dataset) for languages with multiple variants! Refer the 2024 discourse post.

Mozilla is already trying to support languages with multiple orthography. With the features i have suggested on the main post of this issue, it would enhance the participation of everyone in building together the konkani common voice dataset.

I am trying to plan in advance before all the work of translation and sentence collection is done for the konkani language.

Please send a comment if you are okay with combining the dataset on the grounds that users can choose a "language variant" + "writing system" before they record their voices. With your support Mozilla will hopefully make the appropriate changes..

तुमचें वीचार एकदम गरजेचे आसा.

from common-voice.

@alvynabranches, manager of gom locale on pontoon

from common-voice.

Related to #3266

from common-voice.

Giving two options is not possible on a same dataset is not possible hence there is a separate language created for Devanagari and Roman scripts. If this has to be created, we would have to make a custom platform for it. @chasingdragonflies

from common-voice.

If the datasets cannot be merged, then at least the sentences can be linked. It would provide an easier way to further use both datasets for transliteration with machine learning. I am not aware of other use cases at the moment.

How?
In the roman dataset, in sentences.tsv, a new column can be added "devanagari_sentence_id". This can be the sentence ID of the devanagari sentence in devanagari dataset that matches the roman sentence..

Same for devanagari dataset, "roman_sentence_id".

from common-voice.

Tools that can aid in transliteration and linking of sentences:
• https://konkanverter.com, developed by World Konkani Center in Mangalore (automatic transliteration tool, although it's not perfect)
• English to Devanagari-konkani dictionary (official Goa Konkani Basha Mandal app co-developed with shabdkosh)
• Modern English to Roman-konkani dictionary, a PDF file uploaded to wikimedia commons.

from common-voice.

Transliteration of konkani is not as simple as converting devanagari words to english pronunciation.

Here are some examples for the understanding of non-konkani speakers. (Use hindi text-to-speech for pronunciation)

1. For "sweet taste":
   In gom, we say "ghod" (pronunciation: घोड)
   In knn, we say "ghad" (pronunciation: घड)

2. For "my":
   In gom, we say "mojea" (pronunciation: म्हौज्या)
   In knn, we say "mhajea" (pronunciation: म्हज्या)

However, there are some words which have same pronunciation in both konkani scripts..
3. For "is happening":
In gom, we say "zata"
In knn, we also say "zata" (pronunciation: जाता)

from common-voice.

Mozilla common voice can introduce a new tab, "Link", alongside the speak, listen, write, review tabs for the purpose of linking the devanagari and roman sentences across the dataets. This will also boost sentence collection for both datasets

The flow could be: Write -> Review (linguistically correct and copyright-free?) -> Link (type the corresponding devanagari/roman sentence) -> Record -> Validate

Once linked, the other language script can have the flow:
Review (linguistically correct?) -> Record -> Validate.

This way we can collect sentences for both the datasets together!

In the contribution guidelines, for linking purpose, we can suggest the tools i mentioned here which will make it easier to convert the sentences between the devanagari and roman scripts.

from common-voice.

@alvynabranches

Giving two options is not possible on a same dataset is not possible hence there is a separate language created for Devanagari and Roman scripts. If this has to be created, we would have to make a custom platform for it. @chasingdragonflies

They have planned it. Refer: #4144 (comment)

@ginamoape Could you guide us?

from common-voice.

@anniedhempe

from common-voice.

Last date is 25 april 2024

from common-voice.