Question about evaluation: mean rank and mAP about poincare-embeddings HOT 10 CLOSED

I splitted the noun_closure.csv into training and test set, I run embed.py on training set and run my evaluation code on test set to test the model. Am I doing it right?

Assuming there aren't any root/leaf nodes in your validation/test sets, then yes.

When comparing two different models, sometimes one got higher mean rank and higher MAP while the other got lower mean rank and lower MAP. Intuitively, when a model get lower mean rank, it should get higher MAP at the same time. Does this relation always hold?

This relation does not necessarily hold, but is often the case.

from poincare-embeddings.

Rank refers to ranking the neighbors of a given node against it's non-neighbors in the embedding space. A lower mean rank is better, whereas a higher mAP rank is better.

Or maybe there is something wrong with my evaluation code :(

This evaluation should take place after each training epoch. Why do you need to reimplement this?

from poincare-embeddings.

Thanks for your reply.
I didn't figure out how to train on training set while test on test set with train-nouns.sh. I checked the embed.py and it seems that the code trains and evaluates on the same dataset, so I reimplement the evaluation code to run on test data. After spliting the noun_closure.csv into training and test set, I run embed.py on training set and run my code on test set to test the model. Am I doing it wrong?
When I test different models, sometimes mean rank is lower and mAP rank is higher, which indicates better capacity of inferring. However when comparing two different models, sometimes one got higher mean rank and higher MAP while the other got lower mean rank and lower MAP. Intuitively, when a model get lower mean rank, it should get higher MAP at the same time. Does this relation always hold?

from poincare-embeddings.

Assuming you are trying to reproduce the results of Table 1 in this paper, it is correct that you should evaluate on your training set. The point of this evaluation is to show the differences in capacity of the models by seeing how well we are able to represent the original graph in the embedding space.

from poincare-embeddings.

Thanks for the quick reply.
I am trying to reproduce the WordNet Link Prediction results of Table 1.
Should I use embed.py and evaluate on my training set?

from poincare-embeddings.

Correct.

from poincare-embeddings.

So both the Link Prediction and Reconstruction result in table 1 are from evaluation result on training set?

from poincare-embeddings.

Ahh sorry, I wasn't looking carefully enough and thought Table 1 was just reconstruction (and didn't include link prediction). The reconstruction portion of Table 1 uses the entire graph for both training and evaluation. The link prediction does split into train/validation/test sets. Note that as mentioned in the paper:

The validation and test set do not include links
involving root or leaf nodes as these links would either be trivial or impossible to predict reliably.

from poincare-embeddings.

Sorry to bother you again and thanks for your patience. I still got two questions below:

1. I splitted the noun_closure.csv into training and test set, I run embed.py on training set and run my evaluation code on test set to test the model. Am I doing it right?
2. When comparing two different models, sometimes one got higher mean rank and higher MAP while the other got lower mean rank and lower MAP. Intuitively, when a model get lower mean rank, it should get higher MAP at the same time. Does this relation always hold?

from poincare-embeddings.

Got it. Thanks a lot!

from poincare-embeddings.