jonsafari / perstem Goto Github PK

I am executing the script using perl perstem.pl < input > output. input is a file containing the single line

کتابهاصفحه ها صفحهی صفحه ی کار کن کارکن

I would expect many changes to be made to this line through running a stemmer on it, however the output file contains the exact same data as the input data (the diff shell command shows no difference between the two files). I have also tried out all the relevant options (-s --irreg-stem -t 1) to no avail. I have also tried the perl perstem.pl < input | cat > output command to execute the script, but the result was the same.

Hi,
I'm trying to use your stemmer on a linux machine. I run your stemmer using perl perstem.pl input > output where input is the name of the input file containing this single line:

کتابهاصفحه ها صفحهی صفحه ی

However after running the script, I get the following results in the output file:

+�+C+z+A+�+z+N+++F+� +�+z +N+++F+��� +N+++F+� ��

So far as I could tell, this output does have the utf8 encoding however they are somehow distorted.

I would really appreciate it if you could get back to me on this as soon as you can. I could also fix it myself in case there really is a problem and you point me in the right direction.