1. update word-analogy.cpp to use the new mulQuadForm code. The old code is deprecated and should be removed from word-analogy.cpp. To test that this is correct, change the function

void setupDotContainmentMat(int n, real *m) {                                   
    const int d = log2(n);                                                      
    assert (1 << d == n);                                                       
    for(int i = 0; i < n; ++i)                                              
        for(int j = 0; j < n; ++j)
           m[i*n+j] = i == j ? 1 : 0;                                      
}                                                                               

we recover word2vec.

Setup our codebsse such that we can use the gensim.word2vec implementation as well.

• Word2Vec Naive Run
• W2V + Actual SGD (steal from GloVe)
• W2V - 0 initialized + random initialized (Syn1Neg is random :P)
• W2V where syn0 = syn1neg = random
• Train cost function v.w, not sigm(v.w)
• Objective function: cos(theta), not sigm(v.w)

Switch out the dot product of word2vec-ga to be like word2vec. Make a commit with this change, and train the model. Check if the model reproduces word2vec. If not, let's sit together and debug.

Plot vector length against word frequency. Since the word2vec-sn-reimann did not do well, this seems to indicate that training for dot-product 1 does worse than the regular word2vec, for whatever reason. Understand why

let vec(w) . vec(w') = 1.0 - delta(neighbor(w), neighbor(w')). If the neighbourhood structure of w and w' are the same, then we should make them dot product 1, otherwise we should make them dot product based on how close their neighborhoods are.

Running on branch alok-evaluation, as branch master does not seem to have a models directory.

Command: python3 run.py eval models/working-word2vec-baseline-b0c2872c

Output:

setting up device...
device: cuda:1
====time: 0:00:00.616033
loading model from: models/working-word2vec-baseline-b0c2872c...
Traceback (most recent call last):
File "run.py", line 758, in <module>
 PARAMS = torch.load(PARSED.loadpath, map_location=DEVICE)
 File "/home/alokdebnath5/.local/lib/python3.5/site-packages/torch/serialization.py", line 367, in load
	return _load(f, map_location, pickle_module)
File "/home/alokdebnath5/.local/lib/python3.5/site-packages/torch/serialization.py", line 538, in _load
	result = unpickler.load()
AttributeError: Can't get attribute 'Word2Man' on <module '__main__' from 'run.py'>

`
Need help in figuring out why this is happening?

Basically we suspect that even tho the training occurs on 300 dimensions, the actual structure is captured by a much lower dimensional space.
In particular, we should look at conformal dimensionality reduction methods i.e. dimensionality reduction methods which preserve angles.

Need to identify and consider the changes in the code of word2vec to store syn1neg as well, and also to load syn1neg while doing distance and word analogy in order to capture the true meaning of contextualized similarity.

Issue 1: Eval versus wordnet eval in the parameters are interchangably used
Issue 2:

setting up device...
	device: cuda:3
====time: 0:00:00.983759
loading corpus: text8...
Traceback (most recent call last):
  File "run.py", line 1123, in <module>
    main()
  File "run.py", line 1076, in main
    PARAMS = Parameters.load_model_state_dict(LOGGER,
NameError: name 'Parameters' is not defined

https://arxiv.org/pdf/1412.6623.pdf

This is an ICLR 2015 paper that talks about Gaussians for a word representation. We need to use their evaluation section for out paper.

Figure out what downstrem evaluation task is to be used to validate our models

The kind of optimisation that glove tries to perform (finding vectors whose dot products are known) is a well known type of program as far as I understand: In particular, it forms a semidefinite program (SDP)

Here are the possible scenarios:

Glove is SDP

In this case, we use a solver for SDP against their corpus of co-occurences.

We get embeddings better than GloVe ( $$$)

Publish, show why NNs + gradient descent are unnecessary for solving SDPs.

We get embeddings worse that GloVe ( $$$)

There are two reasons for this. Either:

1. Their gradient descent mechanism finds vectors whose dot products better match the co-occurence matrices than a solver for SDP. In this case, we publish in the SDP literature about how gradient descent is better than a fucking solver (highly unlikely) $$$

2. our model has dot products that fits the co-occurence ratios better than glove, but still performs worse ( $$$)

This opens up a line of investigation, since it shows that their "explanation" is BS. Because our model literally optimizes what they claim they are optimizing better, and yet performs worse on the results. This allows us to argue one of:

1. The objective is ill phrased.
2. They are learning something else and their explanation is BS.
3. Their testing regime is BS.

Glove is not SDP for some technical reason :((

We try to massage it into an SDP, but this is the most disappointing possibility

We need to look into the SoTA word embeddings evaluation testbench, in order to analyze the our performance for vectors in downstream tasks

• Fill a 1D array with the value 10
• Fill a 1D array with A[i] = i
• Fill a 2D array with A[i][j] with i*j.
• Vector dot product, simple version, with atomicAdd(&result[i], a[i] + b[i])
• Vector dot product with parallel reduction NVIDIA documentation here
• Perceptron training for w^tX. Parallelise on mini-batches?
• implement word2vec
• implement transformer networks
• implement a 3D physically based rendering engine
• implement a 3D blackhole simulator, and use the rendering engine to raytrace a black hole

Command: python3 run.py train --savepath ./models/

Error:

setting up device...
	device: cuda:3
====time: 0:00:01.179288
loading corpus: text8...
Traceback (most recent call last):
	File "run.py", line 1186, in <module>
	main()
File "run.py", line 1167, in main
	traintype=PARSED.traintype)
File "run.py", line 661, in __init__
	self.NDOCS = int(NDOCS)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' ``

Please explain this as well. Thanks

• Mean L1 loss
• Mean L2 loss

Hey,

Please implement the fill in the blank based evaluation method: I ate a ___ on the bus: Guess the ___ based on the average of the vectors around it.