Link to TensorFlow return an error
"Error fetching projector config. Cannot fetch projector config, possibly a Cross-Origin request error."

https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/akutuzov/824c3429a49948c22d2deb330763f46b

There are still some language-specific strings in templates, for instance, placeholders.
They should be moved to the strings.csv

In the section to the right at http://vectors.nlpl.eu/explore/embeddings/en/calculator/ where you can do algebraic operations on vectors, it is not possible to calculate the "opposite" of a word by just entering that word in the negative field. I was curious to see e.g. what the opposite of dark_ADJ could be.

One way to get around this was adding a dummy word, e.g. boy_NOUN, in both the positive and negative field. The opposite of dark_ADJ, however, was not quite what I had expected. ;-)

Thanks a lot for this WebVectors site! I will take a closer look at the data and the coding when I find some time.

Sometimes proper names are upper-cased when visualizing.
If the model's vocabulary is actually lower-cased, it results in broken links.

"When users use the tool to calculate the similarity between two words, it would be better if the computing history could remain, so users will have a better idea to compare the index of similarity, instead of a single decimal."

Setting interface_languages = en" doesn't have to seem any effect on the frontend, it is still ru/en
@ElizavetaKuzmenko

Здравствуйте, пытаюсь использовать ваши модели в связке gensim,

from gensim.models import KeyedVectors

w2v_fpath = "news_upos_cbow_600_2_2018.vec.gz"

w2v = KeyedVectors.load_word2vec_format(w2v_fpath, binary=False, unicode_errors='ignore')
w2v.init_sims(replace=True)

for word, score in w2v.most_similar(positive=["депутат"], topn=5):
    print(word, score)`

все достаточно просто, на на выходе я получаю ошибку

Traceback (most recent call last): File "/Users/avorontsov/projects/python/gensim/main.py", line 8, in <module> for word, score in w2v.most_similar(positive=["депутат"], topn=5): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gensim/models/keyedvectors.py", line 531, in most_similar mean.append(weight * self.word_vec(word, use_norm=True)) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gensim/models/keyedvectors.py", line 452, in word_vec raise KeyError("word '%s' not in vocabulary" % word) KeyError: "word 'депутат' not in vocabulary"

но если я меняю словарь на один из этих http://panchenko.me/data/dsl-backup/w2v-ru/

то начинает находить. Что я делаю неправильно?

Inline HTML comments describing what's happening in the templates are great.
Unfortunately, some of them are located within loops (example).
As a result, in the final web page, these comments are repeated, sometimes dozens of times. This increases page size and makes its code less readable.

I think, there should be no inline comments within loops. Instead, they can be moved to the beginning of the page, etc.

This mapping do not support noun of mystem lib.
In mystem lib noun is marked by 'S' key.
docs: https://yandex.ru/dev/mystem/doc/grammemes-values-docpage/

@evezhier It would be great to to have JS visualizations of associates' interrelationships.

The number of the retrieved nearest neighbors must be present in in the frontend2backend queries, not hardcoded into the backend (word2vec_server.py).

Templates always transform the words as if they have PoS tags, even if the tags variable in the config file is set to False.
For example, https://github.com/akutuzov/webvectors/blob/master/templates/associates.html#L78

The behavior should be flexible and dependent on the config file.

Hi! Thank you for the great tool!

I have found some strange PROPN usage in preprocessing.
In https://github.com/akutuzov/webvectors/blob/master/preprocessing/rus_preprocessing_udpipe.py#L182 and below it has additional space character in the end.
Maybe it shouldn't have additional space and look like + '_PROPN', not + '_PROPN '?

Also udpipe returns strange result for word "Спасибо":

>>> up.tokenize_upos("Спасибо")
['спасибо_PROPN ']
>>> up.tokenize_upos("спасибо")
['спасибо_NOUN']
>>> up.tokenize_upos("Газпромбанк")
['газпромбанк_PROPN ']

Greetings,

I'm a computer engineering student from Turkey and currently working on creating a Turkish version of Webvectors. However, I encounter a problem. When I click on a word, it attempts to redirect me to a link where similar words are found for that specific word. For example, when I click on the word "adam", it attempts to open an URL like this: http://127.0.0.1:5000/en/word2vec_skipgram_300/adam/. The name of my model is correct. The problem is that there is no such page in the current version of the repository. So I couldn't find an easy way to redirect users to a similar words page for the word "adam". Am I missing anything here?

Let me provide screenshots as well:

1. Here I hover over the word "araba":

2. This is the address it redirects me to upon clicking on it:
   

The 'most similar' block should go to the main page. Its place will be taken by the similarity calculation, and its place in turn will be taken by the 'doesn't match' block

I downloaded ruwikiruscorpora_upos_skipgram_300_2_2019, extracted it and try add to webvectors.

whats incorrect in my configs?

Change font color for a more contrast one.

Коллеги, добрый день. Спасибо за проект. Увидел неточность, возможно стоит исправить.
В файле webvectors/preprocessing/rus_preprocessing_udpipe.py написано "зеленый_NOUN трамвай_NOUN", правильно "зеленый_ADJ трамвай_NOUN" (ваша модель тоже так считает :) ).

Some scripts are not python3-compatible.
For instance, tsne.py with outdated prints and word2vec_server with outdated excepts.

In visualization page, when some words from the query are unknown to the model, the colors for the remaining words are "shifted", leading to incorrect labeling,

If a user selects a model (or a set of models), this choice should persist after the results are shown.
For now, it is reset to the default.

https://pythonclock.org/

Понимаю что ерунда, но все же напишу.
Если выполнять туториал на windows машине(наверное. это странно, да) то в 3 ячейке

text = open(textfile).read()

будут проблемы с открытием файла из-за использования кодировки cp1251 по умолчанию

Может так будет приятнее?

text = open(textfile, encoding='utf-8').read()

Загружаются только через KeyedVectors(), а через gensim.models.fasttext.FastText.load( не загружается
Это воспроизводится в версиях генсима 3.7, 3.6, 3.4

I try run word2vec_server.py. In logs its show that is started. I try connect to port 12666 with telnet - all works. But when i try connect to tagged_port 66666 - its no connection

root@vmi342368:/var/www/webvectors# python3.7 word2vec_server.py
2021-03-19 04:05:00,333 : INFO : loading projection weights from /var/www/model/model.bin
2021-03-19 04:05:10,794 : INFO : loaded (248978, 300) matrix from /var/www/model/model.bin
2021-03-19 04:05:10,794 : INFO : precomputing L2-norms of word weight vectors
Model ruwikiruscorpora_upos_skipgram_300_2_2019 from file /var/www/model/model.bin loaded successfully.
Socket created
Socket bind complete
Socket now listening on port 12666

[Languages]
interface_languages = ru,en

[Files and directories]
temp = tmp
l10n = strings.csv
models = models.tsv
image_cache = images_cache.csv
font = /var/www/webvectors/fonts/OpenSans-Bold.ttf
root = /var/www/webvectors/

[Sockets]
host = 0.0.0.0
port = 12666
tagger_port = 66666

[Tags]
use_tags = True
detect_tag = True
tags_list = ADJ ADV INTJ NOUN PROPN VERB ADP AUX CCONJ DET NUM PART PRON SCONJ PUNCT SYM X
exposed_tags_list = tags.tsv

[Other]
url = /
dbpedia_images = true
default_search = https://duckduckgo.com/?q=
tensorflow_projector = false
git_username = YOUR_GITHUB_USERNAME
git_token = YOUR_GITHUB_API_TOKEN

When clicking on the input field, the provided default value should clear (use placeholder instead of value for a start?)

Prepositions like po, v are excluded from consideration, then some tokens make up mwe. That means that prepositional mwe such as по принципу, в принципе are tagged принцип_NOUN in texts. What is worse, в течение is probably the same as течение_NOUN.
The same issue concerns mwes for conjunctions and particles, and, to a lesser extent, adverbial mwes (or it's another issue).
(If I am wrong and they are filtered out, where can I find their list?)

В зависимостях написать, какие версии генсима, фласка и матплотлиба нужны для работы.

I unzipped the archive and realized that geowac_lemmas_none_fasttextskipgram_300_5_2020 isn't a binary model, that is only 'model.model' was in it and no 'model.bin' like in a ruscorpora_upos_skipgram_600_10_2017 for instance. So, I tried this:

import zipfile
model_url = 'http://vectors.nlpl.eu/repository/20/213.zip'
m = wget.download(model_url)
model_file = model_url.split('/')[-1]
with zipfile.ZipFile(model_file, 'r') as archive:
stream = archive.open('model.model')
model = FastText.load_fasttext_format(datapath(stream))

TypeError Traceback (most recent call last)
in <cell line: 7>()
7 with zipfile.ZipFile(model_file, 'r') as archive:
8 stream = archive.open('model.model')
----> 9 model = FastText.load_fasttext_format(datapath(stream))

2 frames
/usr/lib/python3.10/genericpath.py in _check_arg_types(funcname, *args)
150 hasbytes = True
151 else:
--> 152 raise TypeError(f'{funcname}() argument must be str, bytes, or '
153 f'os.PathLike object, not {s.class.name!r}') from None
154 if hasstr and hasbytes:

TypeError: join() argument must be str, bytes, or os.PathLike object, not 'ZipExtFile'

Then I unzipped the archive, downloaded 'model.model' and 'model.model.vectors.npy' into a google.colab and tried to open each of them direclty eithe via KeyedVectors.load() or via Fast Text.load_fasttext_format()or via gensim.models.fast text.load_facebook_model() or via gensim.models.fast text.load_facebook_vectors().

model = KeyedVectors.load('model.model')

FileNotFoundError Traceback (most recent call last)
in <cell line: 1>()
----> 1 model = KeyedVectors.load('model.model')

4 frames/usr/local/lib/python3.10/dist-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size)
425 own_fid = False
426 else:
--> 427 fid = stack.enter_context(open(os_fspath(file), "rb"))
428 own_fid = True
429

FileNotFoundError: [Errno 2] No such file or directory: 'model.model.vectors_vocab.npy'

model = KeyedVectors.load('model.model.vectors.npy')

UnpicklingError Traceback (most recent call last)
in <cell line: 1>()
----> 1 model = KeyedVectors.load('model.model.vectors.npy')

1 frames/usr/local/lib/python3.10/dist-packages/gensim/utils.py in unpickle(fname)
1459 """
1460 with open(fname, 'rb') as f:
-> 1461 return _pickle.load(f, encoding='latin1') # needed because loading from S3 doesn't support readline()
1462
1463

UnpicklingError: STACK_GLOBAL requires str

model = FastText.load_fasttext_format('model.model')

NotImplementedError Traceback (most recent call last)
in <cell line: 1>()
----> 1 model = FastText.load_fasttext_format('model.model')

5 frames/usr/local/lib/python3.10/dist-packages/gensim/models/_fasttext_bin.py in _load_vocab(fin, new_format, encoding)
196 # Vocab stored by Dictionary::save
197 if nlabels > 0:
--> 198 raise NotImplementedError("Supervised fastText models are not supported")
199 logger.info("loading %s words for fastText model from %s", vocab_size, fin.name)
200

NotImplementedError: Supervised fastText models are not supported

model = FastText.load_fasttext_format('model.model.vectors.npy')

NotImplementedError Traceback (most recent call last)
in <cell line: 1>()
----> 1 model = FastText.load_fasttext_format('model.model.vectors.npy')

5 frames/usr/local/lib/python3.10/dist-packages/gensim/models/_fasttext_bin.py in _load_vocab(fin, new_format, encoding)
196 # Vocab stored by Dictionary::save
197 if nlabels > 0:
--> 198 raise NotImplementedError("Supervised fastText models are not supported")
199 logger.info("loading %s words for fastText model from %s", vocab_size, fin.name)
200

NotImplementedError: Supervised fastText models are not supported

model = FastText.load_facebook_model('model.model')

AttributeError Traceback (most recent call last)
in <cell line: 1>()
----> 1 model = FastText.load_facebook_model('model.model.vectors.npy')

AttributeError: type object 'FastText' has no attribute 'load_facebook_model'

Should I try datapath from gensim.test.utils or api.load from gensim.downloader or pip install fasttext instead of gensim's FastText?
How to download any pretrained fasttext model from rusvectores?

In addition to the tSNE projections, we need PCA projections as well. They might be not so visually attractive, but they are deterministic and stable.

Show word frequencies (and its frequency range), taken from the training corpus.
For that, we would need to load models in Gensim format.

Add option to get similarity results via API (tsv, json).

Now word2vec server and the main code communicate using a strange dialect of strings with separators. Should be brought to normal JSON serialization.

Replace all calls to the deprecated model.wv.vocab with model.vocab, as per changes in Gensim 3.3.0

We should fix it.