According to previous issue issue 57, we propose to add a new function to unshape this text

Salam,
I tested the given words with pyarabic word as follow,
the word contains encoded glyphs not standard letters, it must be converted to ordinary letters.

To convert glyph based word into a string of letters you can use:
NB: the second unshape function is used only to inverse the result word

 word = "ﻣﺴﺎﻣﻌﻬﻢ"
 from pyarabic.unshape import unshaping_word
unshaping_word(unshaping_word(word))
'مسامعهم'

• The test used to detect the problem

``>>> import pyarabic.araby as ar

lst=["اﻟﻤﺴﺌﻮﻟﻴﺔ","ﻣﺴﺎﻣﻌﻬﻢ","ﻓﻜﻠﻨﺎ","ﻣﺒﺎدراﺗﻨﺎ","ﻓﻬﻢ","اﻟﻤﻨﻈﻮﻣﺔ"]
for i in lst:
... print(i, ar.is_arabicword(i))
...
اﻟﻤﺴﺌﻮﻟﻴﺔ False
ﻣﺴﺎﻣﻌﻬﻢ False
ﻓﻜﻠﻨﺎ False
ﻣﺒﺎدراﺗﻨﺎ False
ﻓﻬﻢ False
اﻟﻤﻨﻈﻮﻣﺔ False

for i in lst:
... print("%s"%i, ar.is_arabicword(i))
...
اﻟﻤﺴﺌﻮﻟﻴﺔ False
ﻣﺴﺎﻣﻌﻬﻢ False
ﻓﻜﻠﻨﺎ False
ﻣﺒﺎدراﺗﻨﺎ False
ﻓﻬﻢ False
اﻟﻤﻨﻈﻮﻣﺔ False
for i in lst:
... for c in i :
... print(c, ord(c), ar.name(c))
...
ا 1575 ألف
ﻟ 65247
ﻤ 65252
ﺴ 65204
ﺌ 65164
ﻮ 65262
ﻟ 65247
ﻴ 65268
ﺔ 65172
ﻣ 65251
ﺴ 65204
ﺎ 65166
ﻣ 65251
ﻌ 65228
ﻬ 65260
ﻢ 65250
ﻓ 65235
ﻜ 65244
ﻠ 65248
ﻨ 65256
ﺎ 65166
ﻣ 65251
ﺒ 65170
ﺎ 65166
د 1583 دال
ر 1585 راء
ا 1575 ألف
ﺗ 65175
ﻨ 65256
ﺎ 65166
ﻓ 65235
ﻬ 65260
ﻢ 65250
ا 1575 ألف
ﻟ 65247
ﻤ 65252
ﻨ 65256
ﻈ 65224
ﻮ 65262
ﻣ 65251
ﺔ 65172
`

I am happy to see a documentation website for the library

I suggest using sphinx-rtd theme which supports RTL languages

https://sphinx-rtd-theme.readthedocs.io/en/stable/
https://github.com/readthedocs/sphinx_rtd_theme#contributing-or-modifying-the-theme

thank you

is_arabicword is returning false when passing the following words to it
"اﻟﻤﺴﺌﻮﻟﻴﺔ","ﻣﺴﺎﻣﻌﻬﻢ","ﻓﻜﻠﻨﺎ","ﻣﺒﺎدراﺗﻨﺎ","ﻓﻬﻢ","اﻟﻤﻨﻈﻮﻣﺔ"

i'm trying the exemple below but i'm getting the same result as the input text

from pyarabic.araby import normalize_ligature
text = u"لانها لالء الاسلام"
normalize_ligature(text)

i'm getting output : لانها لالء الاسلام instead of "لانها لالئ الاسلام"

And thanks for your help - very helpfull library

Hi,

when installing from pip, from pyarabic import araby result in this error:
File "", line 1, in
File "/home/naruto/Desktop/herok-app/venv/local/lib/python2.7/site-packages/pyarabic/araby.py", line 28, in
from stack import *
ImportError: No module named stack
the site-packages/pyarabic folder only contains this files:
araby.py araby.pyc init.py init.pyc

Hello. I noticed that the version number has been updated to 0.6.8 in the code, but this version has not been released on Pypi.

https://pypi.org/project/PyArabic/

I thought I would share with you a couple of new Python 3 features that be useful for this codebase. I know that this codebase currently supports Python 2, but Python 2 is now no longer supported by core developers, so this may be the time to drop support for Python 2 and start using these newer features.

This is just a friendly message, I am not expecting you to have the same opinion as me or to prioritise this. Treat this as a conversational message, not as a bug report. Feel free to close this if you intend to keep on supporting Python 2.

\N{name} in Python string literals:

Python 3.3 supports this new feature, see release notes:

Added support for Unicode name aliases and named sequences. Both unicodedata.lookup() and '\N{...}' now resolve name aliases, and unicodedata.lookup() resolves named sequences too.

For example:

>>> "\u0649" == "\N{arabic letter alef maksura}"
True

We can tidy up the codebase to avoid using the hard to read \u0649 escape sequences.

\N{name} in Python regexes:

Python 3.8 supports these new features, see release notes:

Added support of \N{name} escapes in regular expressions:

>>> notice = 'Copyright © 2019'
>>> copyright_year_pattern = re.compile(r'\N{copyright sign}\s*(\d{4})')
>>> int(copyright_year_pattern.search(notice).group(1))
2019

This would allow us to improve code like this:

ALEF_MAKSURA = u"\u0649"
m = re.match(r"^(.)*[%s]" % ALEF_MAKSURA, word)

to look like this instead:

m = re.match(r"^(.)*[\N{arabic letter alef maksura}]")

This is especially useful when using raw strings literals (string literals that are prefixed with r), as it is not possible otherwise to use \N{...} in a raw string for the intended effect. Raw string literals are often used in regexes.

Example Output 1 (a):
Before - English Keyboard:

Hpf lk hgkhs hglj'vtdkK Hpf hg`dk dldg,k f;gdjil Ygn ,p]hkdm hgHl,v tb drt,k ljv]]dk fdk krdqdk>

After:

أحب من الناس المتطرفين، أحب الذين يميلون بكليتهم إلى وحدانية الأمور فلا يقفون مترددين بين نقيضين.

Example Output 2:
Before:
ِىغ هىفثممهلثىف بخخم ؤشى ةشنث فاهىلس لاهللثق ةخقث ؤخةحمثء شىي ةخقث رهخمثىفز ÷ف فشنثس ش فخعؤا خب لثىهعس شىي ش مخف خب ؤخعقشلث فخ ةخرث هى فاث خححخسهفث يهقثؤفهخىز

After:
Any intelligent fool can make things ghigger more complex and more violent. It takes a touch of genius and a lot of courage to move in the opposite direction.

Albert Einstein

References

• Ar-PHP

Hello dear

I'm trying to use pyarabic to tokenize sentences, but I get the bellow error:

AttributeError: module 'pyarabic.araby' has no attribute 'sentence_tokenize'

most of the number words found in Arabic text is usually normalized as Arabic speakers use words like "الف" instead of "ألف" and since the modulo already includes various normalization methods it is saner to use the normalized version of the word instead of the original form

from pyarabic.normalize import normalize_searchtext
normalize_searchtext("بحث")
throws

1. import errors

2. throws typo errors

Add tokenize sentences

Normalizing different digit styles as
توحيد الأرقام وتنميطها إلى شكل معين

• Arabic western numeral: '0123456789'

• Arabic eastern digit: '٠١٢٣٤٥٦٧٨٩'

• Arabic eastern digit variant: '۰۱۲۳۴۵۶۷۸۹'
  الوظائف functions

• Normalize digits to Arabic western تنميط الأرقام للتنويعة العربية المغربية

  • Arabic eastern digit: '٠١٢٣٤٥٦٧٨٩' ==> '0123456789'
  • Arabic eastern digit variant: '۰۱۲۳۴۵۶۷۸۹' ==> '0123456789'

• Normalize digits to Arabic eastern تنميط الأرقام للتنويعة العربية المشرقية

  • Arabic western numeral: '0123456789' ==> '٠١٢٣٤٥٦٧٨٩'
  • Arabic eastern digit variant: '۰۱۲۳۴۵۶۷۸۹' ==> '٠١٢٣٤٥٦٧٨٩'

• Normalize digits to Arabic eastern variant: تنميط الأرقام للتنويعة العربية المشرقية المستعملة في شبه القارة الهندية وإيران

  • Arabic western numeral: '0123456789' ==> '۰۱۲۳۴۵۶۷۸۹'
  • Arabic eastern digit: '٠١٢٣٤٥٦٧٨٩' ==> '۰۱۲۳۴۵۶۷۸۹'

As-salamo Alaikom Taha @linuxscout ,

I am testing the removal of dots above or below Arabic letters, and everything seems to be working fine, except for initial and middle noon (حرف النون).

Any idea if there is a fix for this?

Here is a screen shot.

Thanks

Great work Taha. Have you considered adding stop words removal support? There's a good list here that you could start with. https://github.com/mohataher/arabic-stop-words

It came with this error:
NameError: name 'is_arabicstring' is not defined
only is_arabicrange is working well

Salam Dr.Taha, 👋

There is an ImportError when importing module number.py 😕

To reproduce

>>> import pyarabic.number as number 

Error Report

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "\venv\lib\site-packages\pyarabic\number.py", line 27, in <module>
    from six import text_type as unicode
ModuleNotFoundError: No module named 'six'

My System Version

• Python 3.8.6

Temporary fix
Installing package six using pip install six

Suggestions
I suggest adding six "current version" into the dependencies of pyarabic within install_requires
I'm relatively new to Python "couples of months" and this suggestion is based on my humble experience with Python. 🥺
This is also my first issue submission into GitHub so please bear with me! 😅

The strip_tashkeel and strip_diactricts functions are very helpful when preprocessing text that will be used for searches. With these functions, one can search for a word that like رحيم without tashkeel. However, one of the challenges is this will not match a word that had tanween al fateh at the end, as the word after removing the tashkeel will still be different in structure رحيما.

I suggest adding another optional flag (to support previous versions) that will also remove the alef if it comes after tanween al fateh. See https://en.rattibha.com/thread/1266046390439903234 for details

Thank you for the amazing library!

Hi, I am an Urdu speaker. I am trying to convert arabic words to simple words . I found the stated behavior, please correct me if i am wrong , may be it is intended or correct behavior. If that's the case how can i not change last YEH (small etc) to alef.

words = ['إِلَّا','إِلَىٰ','بِى','بِٱلْهُدَىٰ','بِٱلَّذِىٓ','بِلِقَآئِ','بُغِىَ','ٱلْمَأْوَىٰ']
for w in words:
    simple=w
    print(f'word : {simple}')
    
    simple=araby.strip_diacritics(simple)
    print(f'strip_diacritics : {simple}')
    
    simple=araby.normalize_alef(simple)
    print(f'normalize_alef : {simple}')
    
    simple=araby.normalize_hamza(simple)
    print(f'normalize_hamza : {simple}')
    
    print('_'*10)

outputs:

word : إِلَّا
strip_diacritics : إلا
normalize_alef : الا
normalize_hamza : الا
__________
word : إِلَىٰ
strip_diacritics : إلى
normalize_alef : الا
normalize_hamza : الا
__________
word : بِى
strip_diacritics : بى
normalize_alef : با
normalize_hamza : با
__________
word : بِٱلْهُدَىٰ
strip_diacritics : بٱلهدى
normalize_alef : بالهدا
normalize_hamza : بالهدا
__________
word : بِٱلَّذِىٓ
strip_diacritics : بٱلذى
normalize_alef : بالذا
normalize_hamza : بالذا
__________
word : بِلِقَآئِ
strip_diacritics : بلقائ
normalize_alef : بلقائ
normalize_hamza : بلقاء
__________
word : بُغِىَ
strip_diacritics : بغى
normalize_alef : بغا
normalize_hamza : بغا
__________
word : ٱلْمَأْوَىٰ
strip_diacritics : ٱلمأوى
normalize_alef : الماوا
normalize_hamza : الماوا

The README.md file points to this site for documentation:

https://pythonhosted.org/PyArabic/

Unfortunately, it currently is broken. It does not display properly on Chrome or Firefox. If I look in the console, I see these errors:

Load denied by X-Frame-Options: https://pythonhosted.org/PyArabic/toc.html does not permit framing.

I'm guessing pythonhosted.org recently turned on the X-Frame-Options: deny header, making it impossible to use iframes the way the current documentation uses iframes.

@linuxscout is there any Sentence Tokenizer function ?

Hi @linuxscout

The package documentation links to https://pythonhosted.org/PyArabic/ but the link appears to be broken. Or is https://pypi.org/project/PyArabic/ the main documentation for the package?

Could you clarify this point, please?

Kamran

Hi, I am wondering if the package supports python 3.x

Hi
is there any function that could provide prefix and suffix tokenization?
such as:
المدخل: ولن نبالغ إذا قلنا: إن 'هاتف' أو 'كمبيوتر المكتب' في زمننا هذا ضروري
المخرج: و+ لن نبالغ إذا قل +نا : إن ' هاتف ' أو ' كمبيوتر ال+ مكتب ' في زمن +نا هذا ضروري

تنظيف وتنميط النص العربي بحذف العلامات المختلفة مثل :

• العلامات القرآنية الموجودة في يونيكود
  • مدخل: -يُنَزِّلُ ٱلْمَلَٰٓئِكَةَ بِٱلرُّوحِ مِنْ أَمْرِهِۦ عَلَىٰ مَن يَشَآءُ مِنْ عِبَادِهِۦٓ أَنْ أَنذِرُوٓاْ أَنَّهُۥ لَآ إِلَٰهَ إِلَّآ أَنَا۠ فَٱتَّقُونِ‎
  • مخرج - يُنَزِّلُ الْمَلَائِكَةَ بِالرُّوحِ مِنْ أَمْرِهِ عَلَىٰ مَن يَشَاءُ مِنْ عِبَادِهِ أَنْ أَنذِرُوا أَنَّهُ لَا إِلَٰهَ إِلَّا أَنَا فَاتَّقُونِ
• علامات الزخرفة
  • مدخل: الہلہغہة الہعہربيہة+ال͠ل͠غ͠ة ال͠ع͠رب͠ي͠ة+الہٰلہٰغة الہٰعربٰٰيٰة+ال̲ل̲غ̲ة ال̲ع̲ر̲ب̲ي̲ة
  • مخرج: اللغة العربية
• علامات الإيموجي
  • مدخل:
  • مخرج:

مراجع:

• Clean Text library
• Tnkeeh
• أداة زخرفة النصوص

This looks promising for Arabic tokenization. Not an issue, but It'll be great to provide options in the tokenizer - ex. remove tashkeel and filter non-Arabic words in a mixed text.

The documentation mentions that the library has a number to text (21 -> واحد وعشرون) method, but I can't find it. Would you please point to it? And give an example of how to use it?

Arabic Text Standardize:

Standardize Arabic text just like rules followed in magazines and newspapers like spaces before and after punctuations, brackets and units etc ...

Example Output:
Origenal:

هذا نص عربي ، و فيه علامات ترقيم بحاجة إلى ضبط و معايرة !و كذلك نصوص( بين أقواس )أو حتى مؤطرة"بإشارات إقتباس "أو- علامات إعتراض -الخ......
لذا ستكون هذه المكتبة أداة و وسيلة لمعالجة مثل هكذا حالات، بما فيها الواحدات 1 Kg أو مثلا MB 16 وسواها حتى النسب المؤية مثل 20% أو %50 وهكذا ...
Standard:

هذا نص عربي، وفيه علامات ترقيم بحاجة إلى ضبط ومعايرة! وكذلك نصوص (بين أقواس) أو حتى مؤطرة "بإشارات إقتباس" أو -علامات إعتراض- الخ...
لذا ستكون هذه المكتبة أداة و وسيلة لمعالجة مثل هكذا حالات، بما فيها الواحدات 1 Kg أو مثلا 16 MB وسواها حتى النسب المؤية مثل %20 أو %50 وهكذا...

Ar-php: text standardize

Is there any plan for Python 3 support ? Any alternative ?

I want to Add a function to correct punctuation for joined texts

example :

def fix_punct(text):
    fix_spaces = re.compile(r'\s*([?؟!.,،]+(?:\s+[?؟!.,،]+)*)\s*', re.UNICODE)
    text = fix_spaces.sub(lambda x: "{} ".format(x.group(1).replace(" ", "")), text)
    return text.strip()
fixed = fix_punct(u"كل فرد في الأمة مجند لمعركة المصير : الفلاح في حقله ، والعامل في مصنعه ، والطالب في معهده ، والموظف في ديوانه ")
print(fixed)
>>> 
كل فرد في الأمة مجند لمعركة المصير : الفلاح في حقله، والعامل في مصنعه، والطالب في معهده، والموظف في ديوانه...

Expected Behaviour

An Arabic word doesn't contain spaces, digits and punctuation

Current Behaviour

araby.is_arabicword('؛') gives True.

Add functions to list letters and their codes and names

In Tokenize part, it didn't separate the character و from the word when it is not a part of the original words, like in the example:

>>> from pyarabic.araby import tokenize, is_arabicrange, strip_tashkeel
>>> text = u"ِاسمٌ الكلبِ في اللغةِ الإنجليزية Dog واسمُ الحمارِ Donky"
>>> tokenize(text, conditions=is_arabicrange, morphs=strip_tashkeel)
        ['اسم', 'الكلب', 'في', 'اللغة', 'الإنجليزية', 'واسم', 'الحمار']

Hi,
Is there any function that checks if a word is Arabic or not?
e.g.
is_arabic('محمد') => True
is_arabic('’Muhammed') => False

Thanks.

setup.py mentions that this is licensed under GPL. However, which version of the GPL? And where is the full text of the license?

I recommend including a file named LICENSE.txt with the full text of GPLv3. Here are instructions: https://choosealicense.com

• spellit

• encode_tashkeel

• normalized_digit

• fix error on project description on pypi with features.md

أربعة مئة = 400 -> giving output as 104
104 = مائة وأربعة -> giving puput as 100 و 4

السلام عليكم، شكرا لك على هذه المكتبة. كما تعلم، سيتوقف دعم بايثون 2 في العام القادم، هل هناك نسخة متوافقة مع بايثون 3؟
جزاك الله خيرا

I updated to 0.6.3 and my application fails with this Traceback:

Traceback (most recent call last):
File "C:\Users\tahoar\slatedesktop\share\plugins\ar\tokenizer.py", line 23, in
from pyarabic import araby
File "C:\Program Files\Python36\lib\site-packages\pyarabic\araby.py", line 219
HARAKAT_PATTERN = re.compile(ur"[" + u"".join(HARAKAT) + u"]", re.UNICODE)
^
SyntaxError: invalid syntax

I have done considerable work on your code to update all modules for Python 2.7 and 3.x compatibility and I'd like to contribute those changed to trunk. Please review this attachment and run through your tests.

Thanks,
Tom

pyarabic.zip

Is there any builtin function to split letter/character?

I just tried installing using pip, but it gave me encoding error in the setup.py file.
To fix this i added encoding='utf-8'.

def readme():
    with open('README.md',encoding='utf-8') as f:
        return f.read()

Convert ordinal number into number
"الثالث" => 3

From what I have read, there is only one kind of text normalization: Hamazat normalization.

What About other letters? like what is in ar-php library.

for example
Origenal Text: آسِفـــةٌ لا تَنَبُّؤْ

Normalized Text: اسفه لا تنبء

Is there anything similar to that? It would be very helpful..

thanks @linuxscout for all you work!

Add a fucntion to remove diacritics like Small alef