Transfer Entropy on Different Dimensions? about entropy_estimators HOT 2 CLOSED

When computing the transfer entropy TE(X->Y), you typically only consider the past states of X and Y at a single lag \tau, with \tau typically being 1. Your first example would then be:

import numpy as np
from entropy_estimators import continuous

np.random.seed(42)
x = np.random.rand(3000)
y = np.random.rand(3000)

# T(X->Y) = PMI(Y_future, X_past, Y_past)
transferXtoY = continuous.get_pmi(y[1:], x[:-1], y[:-1], estimator="fp")

print(transferXtoY)

I think this is what you meant to write in your first example but you didn't quite get the offsets right, as

print(np.all(y[-3000:] == y[0:3000]))
# True

You are completely right that it often would be useful to consider more than a single lag value. If you wanted to condition on the last two values of Y, the code would look like this:

z = np.c_[y[:-2], y[1:-1]]
print(z.shape)
# (2998, 2)
transferXtoY = continuous.get_pmi(y[2:], x[1:-1], z, estimator="fp")

However, just because you can, doesn't mean you should. Note that you have increased the dimensionality of the joint probabilities that (implicitly) have to be estimated. Each additional dimension exponentially increases the number of data points needed to get a good estimate of the local density. This is analogous to the fact that the number of data points to construct a decent histogram increases exponentially with the dimensionality of the data: for a 1D histogram you can maybe get away with 30+ points whereas for a 2D histogram you probably need 1000+, i.e. 30^2 points.

There are alternative approaches that are often more fruitful. The simplest option, in my opinion, is to impose a model on how past states of Y influence future states of Y. For example, instead of conditioning on the last 10 values of Y (thus adding 10 dimensions), you could condition on the (weighted) sum of the last 10 values (which would again just be a 1D vector).

Having such a condensed representation of a variables history is also preferable for another subtle but important reason: the k-NN estimators are not scale invariant. As a consequence, with the naive approach

z = np.c_[y[:-2], y[1:-1]]
print(z.shape)
# (2998, 2)
transferXtoY = continuous.get_pmi(y[2:], x[1:-1], z, estimator="fp")

you are effectively assuming that states at time t=-2 have an equally strong predictive power on current states as do states at t=-1, whereas in reality, states at t=-1 are often more predictive of the current state than previous states.
Having a more explicit forward model allows you to bake better assumptions into your calculations, but without knowing what data you are working with, it is pretty difficult to advise you on that front.

from entropy_estimators.

Ah, I was missing the lag, thank you!

The note on the data requirements for accurate approximation as a function of the lag is very important, thank you for mentioning. A 1D vector of the weighted sum of the past n values is a great idea for an efficient approximation.

This clarifies the usage for me and is in line with my theoretical understanding of transfer entropy, I'll go ahead and close.

from entropy_estimators.