Diacritics restoration in Text-to-text (Arabic) about dsnote HOT 7 CLOSED

Thanks for the question.

Indeed, "Punctuation restoration" function is definitely not well explained in the app - sorry.

"Punctuation restoration" is an extra text processing after STT. This processing uses additional ML model to guess the right latin punctuation marks (,.?!:) in the text. The model needs to be downloaded via model browser. Right now, it supports only 16 languages (no Arabic support). "Punctuation restoration" is only needed (and only enabled) for DeepSpeech, Vosk and few April models. Whisper and Faster Whisper natively support punctuation therefore this feature is not used for these models. If you are using model from Whisper family, "Punctuation restoration" option (enabled or disables) has not impact on processing.

from dsnote.

Regarding "diacritics restoration". It is implemented but not available for text-to-text conversion. When you mark "Restore diacritics before speech synthesis" (enabled by default) and start TTS, input text is processed to restore diacritical marks in the background and output is sent to TTS engine. Right now, you can't see text after diacritics restoration.

from dsnote.

Whisper and Faster Whisper natively support punctuation therefore this feature is not used for these models. If you are using model from Whisper family, "Punctuation restoration" option (enabled or disables) has not impact on processing.

My understanding of punctuation is possibly wrong here then.

Arabic can come in 2 formats:

فَرَائِضُ الْوُضُوءِ وَفَرْضُهُ: غَسْلُ الْوَجْهِ، وَغَسْلُ الْيَدَيْنِ مَعَ الْمِرْفَقَيْنِ ، وَمَسْحُ رُبُعِ الرَّأْسِ، وَغَسْلُ الرِّجْلَيْنِ مَعَ الْكَعْبَيْنِ. - diacritics

فرائض الوضوء وفرضه: غسل الوجه، وغسل اليدين مع المرفقين ، ومسح ربع الرأس، وغسل الرجلين مع الكعبين. - no diacritics

Diacritics could be considered punctuation (or vowelized). I assumed that the Whisper models would restore the diacritics as a by-product of punctuation, but the output was like the second example (no diacritics) and hence why I opened this ticket.

from dsnote.

Regarding "diacritics restoration". It is implemented but not available for text-to-text conversion. When you mark "Restore diacritics before speech synthesis" (enabled by default) and start TTS, input text is processed to restore diacritical marks in the background and output is sent to TTS engine. Right now, you can't see text after diacritics restoration.

This would be a nice feature to implement. Considering it is already restoring the diacritics for TTS, generating the output for the user could help them see if the model is not hallucinating when reading the text.

from dsnote.

Adding to TO-DO list for the next release.

from dsnote.

Sorry it took so long.

New version 4.5.0 brings following feature:

• Text to Text repair options. With these options you can directly fix diacritical marks and punctuation in the text.

The "repair options" are not enabled by default. You have to turn-on "Show Repair text options" in the settings.

When enabled, you can use this:

You need also download additional model:

30695bb

from dsnote.

Thank you so much for implementing this feature and no need to apologize. I am very grateful that you even considered implementing this feature, which would be of limited usage to most other users.

from dsnote.