valedica / twec Goto Github PK

from twec.twec import TWEC    
from gensim.models.word2vec import Word2Vec

def train():    
    aligner = TWEC(size=30, siter=10, diter=10, workers=4)    
    aligner.train_compass("examples/training/compass.txt", overwrite=False)     
    slice_one = aligner.train_slice("examples/training/arxiv_14.txt", save=True)    
    slice_two = aligner.train_slice("examples/training/arxiv_9.txt", save=True)

def test():    
    model1 = Word2Vec.load("model/arxiv_14.model")    
    model2 = Word2Vec.load("model/arxiv_9.model")    
    for model in [model1,model2]:    
	    print(sum(model.syn1neg))    
	    print(sum(model.wv.syn0))   

if __name__ == "__main__":    
    train()    
    test()

The output of the code is shown as below:

`[ 54.13019 -7458.793 -2588.298 3593.7505 -731.2068 1354.8907
1956.362 2851.0269 -1234.2087 -2461.2375 693.96765 4517.283
1506.449 -1617.4432 1538.4094 2772.7483 2216.757 -3763.828
2090.126 -298.45084 -294.8205 1523.8512 -4156.9824 -723.04803
-533.2238 1869.8455 -1205.959 -3589.7622 -7645.8135 -4966.196 ]

[ 75.728424 2146.7905 711.1423 -1063.1915 280.071
-428.7143 -653.19977 -737.3386 470.85577 737.1261
-51.172543 -1358.5729 -683.6471 417.5251 -398.98938
-808.00616 -600.1352 1040.6033 -659.40375 73.63555
73.206184 -372.51102 1261.4464 297.45206 212.58424
-495.39255 383.86707 955.2797 2138.7588 1448.5309 ]

[ -206.65964 -5041.21 -2035.7019 2772.9456 -725.939 1060.8079
1505.944 2003.1798 -563.0721 -1705.3502 515.3484 3435.2378
1639.7721 -1262.4358 1019.02844 1742.8516 1668.6241 -2807.0754
1269.7594 -494.86893 -221.1095 729.1342 -2732.2847 -153.8587
-501.57608 1336.3754 -1268.0028 -2143.7483 -5006.103 -3494.257 ]

[ 172.6337 2415.6028 1006.8115 -1404.1216 438.4736
-564.31976 -789.87054 -883.25604 373.4988 959.29047
-90.953415 -1708.9927 -1026.617 612.2216 -466.75372
-864.9828 -801.6127 1305.5497 -626.8068 282.45493
129.64682 -274.8585 1347.7399 130.84848 272.3334
-714.29504 643.37933 997.20715 2441.326 1698.4065 ]`

We could clearly see that both the context embedding and target embedding are not fixed. If the compass is not fixed, this work could be very similar to Kim et.al., and all word embedding in different time are not aligned, and therefore it is a little bit risky to compare word vectors in different years especially we train the temporal word vectors with more steps/epochs.

However, in the paper, it was stated in the section of 'Temporal Word Embeddings with a Compass':

During this training process, the target
embeddings of the output matrix U are not modified, while
we update the context embeddings in the input matrix Cti

Am I wrong here? Is there anything I did not notice?

Hi,

thank you for this package! I have just run twec on my data and all has worked well however:

It is unclear on how to access metrics relating to this data. For example, I can access the standard word2vec characteristics that show me most similar embeddings in the individual models (slice_one, slice_two), however I want to start to understand how specific words change in the context of time.

Are there handy functions for exploring most changing words and also for plotting them as in the published paper?

Is there also a way of plotting the data in the same way as figure 1 in:

https://ipg.idsia.ch/preprints/supsi2020a.pdf

This is how i would like to demonstrate how some words change over time, but unsure as to how to plot this data from TWEC model.

thanks!

Add a method analogy(w1, t1, t2) to resolve a temporal analogy w1 : t1 = x : t2.