Remove Diacritics for Urdu Language about texthero HOT 7 CLOSED

I believe the issue is that the proposed changes are on the Fix-Remove-Diacritics branch of the repository. With your command, pip uses the master branch where the changes are not yet implemented.
You can try using !pip install git+https://github.com/SummerOfCode-NoHate/texthero.git@Fix_Remove_Diacritics, I think that should work. If not, you could just copy the functions I pasted in #72 and try them out directly.

from texthero.

Hi, I just had a look and opened a PR to fix this at #72 . I pasted the new functions there, would be great if you could try them out and comment there whether everything works for you. Maybe I missed something regarding the Urdu language.

from texthero.

You can correct me if am doing wrong, I have never engaged into Github.
Installed your version,
!pip install git+https://github.com/SummerOfCode-NoHate/texthero.git
and tested with the same code

import pandas as pd 
import texthero as hero
s = pd.Series("Montréal, über, noël, 889, اِس, اُس")
s1 = hero.remove_diacritics(s)
s1

gives the following output

0    Montreal, uber, noel, 889, is, us
dtype: object

from texthero.

Hi Hashim,

thank you for your contribution!

@henrifroese and @cmhashim, probably the way we should design multilingual support for Texthero is to have:

from texthero.ur import hero
hero.remove_diacritics(...)

Where this remove_diacritics is specialized in dealing with Urdu text.

What's your opinion? That way we can keep the code in each function simple, as well as develop each function for that specific language.

from texthero.

I think that if functions for multilingual support are added (e.g. functions to handle stuff regarding arabic script specifically) they should get separate modules and that would make sense. However, I think that this issue/fix is more generally improving the remove_diacritics by preventing transliteration, so it can now handle everything from before + urdu / arabic / ... , which is why I wouldn't put it in a separate module.

from texthero.

That makes complete sense! 👍

from texthero.

from texthero.ur import hero
hero.remove_diacritics(...)

Where this remove_diacritics is specialized in dealing with Urdu text.

I think its time to do this. It was my mistake i gave very simple example of Urdu text with diacritics, but it much more complex to handle diacritics in Urdu/Arabic. Some diacritics are part of Urdu words, and it must be written, and some can be excluded. Hence, can we have a optional argument, to exclude/include a list of diacritics to retain/remove it.

Some Examples:

retain_diacritics_eg_text = "فوراً, حتیٰ, آزاد, ہوئی"

from texthero.