akoksal / turkish-word2vec Goto Github PK

In readme file, Turkish translation of corpus should be 'derlem' not 'korpus'.

Upload pretrained models with and without lemmatization to google drive.

Deadline: 28.02.2018

I am getting the error below while working on Windows. I believe that it is due to different behavior of built-in multiprocessing library. I will check to handle this issue.

AttributeError: Can't get attribute 'tokenize_tr' on <module '__mp_main__' from 'C:\\Users\\abdullatif.koksal\\Desktop\\Named Entity Recognition\\Word2Vec\\Wiki 2Text\\preprocess.py'>

I was following the instructions in the wiki page to load and use saved models, (Section 5) downloaded the pretrained model but it seems like gensim cannot load the model from file. When I try to load model like so:

from gensim.models import KeyedVectors
word_vectors = KeyedVectors.load("trmodel")

I get an error at the word_vectors = KeyedVectors.load("trmodel") line that says:

AttributeError: Can't get attribute 'EuclideanKeyedVectors' on <module 'gensim.models.keyedvectors' from "DIR_OF_keyedvectors.py_UNDER_ANACONDA_SITE_PACKAGES_FOLDER"'

Here is my setup:

• Python 3.6.4 : Anaconda 64-bit
• Gensim 3.4.0
• Ubuntu 17.10 64-bit

I think it's because versions of gensim used for training and loading the model are different, some functions might be changed or deprecated or something. Could you check it out please?

firstly thanks for sharing this source.

my problem is that you used wikicorpus for opening your corpus but i built mine text without wiki. how could i implement mine ?

wiki = WikiCorpus(inputFile, lemmatize=False,tokenizer_func = tokenize_tr)

Use https://github.com/akoksal/Turkish-Lemmatizer to create model with lemmas.