Stability related and MAR score/Conditional Density calculation issue about know-evolve HOT 8 OPEN

I also faced the same issue. Most sim and curr_sim values are -inf.
I think this is because this->event_t - this->cur_time.GetCurTime(subject, object) = 0 in most cases.

from know-evolve.

hi, Sorry ,i have some questions about understanding the paper at loss function, i do not know why it's survival term need sum in all subject entity , all object entity and all relations

from know-evolve.

As far as I understand, the survival term indicates that no events happen during the time interval between two events(observed). Hence you minimise the probability of the corruptions occurring during the interval.

from know-evolve.

@authors
I played with various hyper parameters and I always get similar output. I guess you are reusing the framework for your other paper as well and you have evaluated the KE framework on other datasets. Can you please verify this issue and please share the right set of hyper parameters where the LogLL doesn't go to infinity.

Currently I am getting excellent results on ICEWS-full, however that's because of the issue that I have mentioned above, in which case, the results would not be correct(as it doesn't rank the positive triplet against any corruption and holds the MAR score at 1 - i.e the if condition doesn't work due to -Infinity comparison)

from know-evolve.

Queries about the paper:

1. Equation 3 for the conditional intensity(lambda) is not bounded in the positive direction. So during optimisation it can explode depending on the gradient it receives (as lambda = exp(g)). How to prevent this from happening?

• While you train over the whole dataset as a single sequence (rather than breaking the sequence as done in conventional RNN), you do the truncated backpropagation and sliding window training using a minibatch. The batch size helps to deal with the issue of vanishing and exploding gradients. The Global BPTT algorithm has more details on the training.

2. In Equation 2, to compute the conditional density of an event, we have f = lambda * S. Aren't these two terms competing against each other? As lambda increases, S decreases. How do we ensure the stability while training?

• This is not true. lambda is the probability that a single event happens in an instantaneous time window [t, t + delta t]. S is the probability that this event survives (hence S is called survival term), that is upto any given future time point t' (t'>t), no other event occurs. These two terms are not competing with each other, the term f is a conditional probability density function, hence, there is no issue of stability in this case.

3. In section 4, you mention a trainable Parameter W_e, but this is not used in any of the equations in the paper. What is this related to?

• Both W_e and W_r refer to the initial projection parameters that are used to project the entity and relation features (e.g. 1-hot vectors ) to low-dimensional embeddings

4. It is also mentioned that V_e is a trainable parameter. Do you mean that it is trainable because it evolves over time, or it is actually a trainable node (i.e. apart from temporal evolution, it gets updated during back prop)?

• V_e is the embedding holder that evolves over time as you have mentioned but not trained through backprop.

• I will respond to code-related queries in a separate response. Please let me know if something is not clear in the above responses.

from know-evolve.

hi, Sorry ,i have some questions about understanding the paper at loss function, i do not know why it's survival term need sum in all subject entity , all object entity and all relations

Consider that lambda signifies the probability of an event happening in an instantaneous interval [t, t + dt]. And the survival term signifies no further event happening upto some future time point t'. However, in this case, the happening or non-happening of event is qualified by a tensor - an event happens between a subject s, object o and of relationship r. And hence probability events not happening should also be computed across all the combinations possible in this tensor. This is why the sum over all three components is required.

from know-evolve.

Hi Rakshit, Any updates on this issue? Have you looked at the rank computation error that I have mentioned above?

from know-evolve.

Hi Everyone,

Thank you for your queries and sorry for latency on my side.

We are aware of this issue and working to address it. As such, this is not due to a code problem but the artifact of dataset. In general, the model assumes continuous time data however, these relational datasets also have several events (involving same subject or object) at single time point. This leads to a situation where you get this->event_t - this->cur_time.GetCurTime(subject, object) = 0 in several samples.

This is not a problem during training as the rank is an auxiliary report and not used anywhere. During evaluation however, following happens: We update the embeddings of entities after each test point. However, in this particular case of multiple events at single time point, for a given test triple (sim), while computing the rank against other objects (cur_sim), the rank should not be affected by the object embeddings that underwent an update in the same time point. It should only consider object embedding that were updated in previous time point. Currently, when the cur_sim is computed for the objects that had an update in same time point, it gives -inf (due to t-t' term) for such objects and thereby ignores that object, essentially ranking only against objects that had got updated previously. However, I am currently working on a better way to handle this case without causing infinity. Please note the infinity issue won't occur if you dont have same entity participating in multiple events at same time point. Also, I found a couple of other minor fixes that needs to be included. I will post here in next 2-3 weeks to provide new updates. Please stay tuned.

Thanks

from know-evolve.