Unicode decode error while using KPMiner about pke HOT 7 CLOSED

Hi @dipaweshpawarsn,

Do you also get that error when using the TfIdf model?

f.

from pke.

I will have to check that as I tried it 3 months ago

from pke.

Hi @dipaweshpawarsn,

Do you also get that error when using the TfIdf model?

f.

No I have not tried it.

from pke.

I am not able to reproduce your issue on my computer. Can you try to update your pke install and re-try?

Thanks,

from pke.

I am able to reproduce this issue in an Ubuntu-18.04 Docker Container with Python 3.6. It works on the host though, which is also 18.04 with python 3.6

from pke.

The fix is to run the following:

apt-get install locales
locale-gen en_US.UTF-8
update-locale LC_ALL= en_US.UTF-8 LANG= en_US.UTF-8
export PYTHONIOENCODING=utf8

Not sure whether all lines are necessary.

from pke.

I fixed the same bug as follows.

File: pke/util.py
Function: load_document_frequency_file(input_file, delimiter='\t'):
Lines: 51 ~ 53
Fixed code: (adding encoding="utf8" when open the file)

    # open the input file
    with gzip.open(input_file, 'rt', encoding="utf8") if input_file.endswith('.gz') else \
            codecs.open(input_file, 'rt', encoding="utf8") as f:
        # read the csv file
        df_reader = csv.reader(f, delimiter=delimiter)

from pke.