nicktfranklin / sem_paper_simulations Goto Github PK

Could do something like particle filtering, where each particle = hypothesis about the event sequence (rather than considering the MAP only).

Could include pruning of unlikely hypothesis (e..g as you accumulate more scenes of the same event type) and resampling at moment of high uncertainty (e.g. at event boundaries when there's lots of potential candidate events). Would address the problem of how do you determine the event type given a single scene after an event boundary. Would also make distinct predictions about order effects (e.g. events that start off similarly but evolve differently -> do people re-assign the initial scenes to the eventually inferred event?).

Alternatively, could do something more naive like only considering the last K scenes (e.g. last 3 scenes) for resegmentation, and only testing the hypothesis that they all belong to the currently active event type vs. that they were segmented correctly the first time. Would again address the issue of segmenting at event boundaries.

Notice this makes a difference both for prediction (it could turn out that we're more likely to be in a bar fight but only if we consider the full history of scenes from the beginning, which we may have assigned to a different event type already, e.g. an academic debate) as well as for training (we should train the bar fight model on those scenes too; conversely, even though we may not be able to "untrain" the academic debate model, we could remove them from its "episodic memory" if we're doing batch gradient descent later on).

It would be nice to have a way to very quickly compare performance of different event models (e.g. RNN vs. stateful RNN vs. LDS vs. LSTM ... etc) on different examples. Ideally, we could have something that automatically (deterministically -- for reproducibility) generates test cases based on different templates (e.g. simple LDS, coffee shop world, the dancing people study, etc) and evaluates different versions of SEM on:

• accuracy of event segmentation
• accuracy of scene predictions
• speed
• how well it recapitulates empirical phenomena

Potentially we could parametrize all of the SEM code to make the process easier -- it will make the code kind of ugly but we will avoid the need to maintain a thousand different files / branches with each separate version.

The prediction appear to be sensitive to the optimization algorithm of the event model. Add these to a test bed.

Requires some investigation.

Say we have the following "true" event sequence: AAAAABBBBBCCCCC, so e1..e5 = A, e6..e10 = B, and e11..e15 = C.
Say we've segmented s1..s10 correctly and now we're at s11.

We have three options for the event type of s11:

1. the current event type (B)
2. a previously seen event type (A)
3. a previously unseen event type (C)

How do we compute the likelihood of s11 in each of these?

Case 1 is straightforward -- we feed the previous scene s10 to the current event model B to get the predicted next scene s11.

Case 2 is not so straightforward, because of the implied event boundary. Several options:

• Same as case 1, except using the event type under consideration (A). This is what we currently do. That doesn't make a lot of sense though -- if we believe there is an event boundary between s10 and s11, why would we want to use s10 to predict s11 (unless we're predicting across events, which is not in the framework but is definitely an interesting question)? Notice that this is not much of an issue once we're at s12. The implication is that we would consistently make incorrect predictions for the first scene after an event boundary, but would quickly correct ourselves.
• Predict the average scene for the event type under consideration. Assumes that scenes from each event type tend to cluster around the same area of scene space.
• Predict the average initial scene for the event type under consideration. Assumes that each sequence of scenes from an event type tends to start from the same area of scene space.
• Feed some "null" scene (instead of s10) to the event model under consideration. If the event model is general enough (e.g. a fancy RNN), it should eventually learn that "null" scene means start of a new sequence and figure out what the best predictions is there.
• Predict some "null" scene

Case 3 is even less straightforward, because we don't have an initialized event model. Options:

• Same as case 1, except using a newly initialized event model with random parameters. Currently we do something of this sort, except we should look more carefully at how we do it for the different event models.
• Feed "null" scene (instead of s10) to newly initialized event model. Risks significantly lowering the probability of picking a new event type (if the initial weights are too constrained).
• Predict the "null" scene.
• Predict the last scene s10. Risks always selecting new event types if you have a sequence of identical scenes (which seems like it should suggest the same event type).

The Jacobian (technically df/dW, not dX/dW) used to update the LDS seems incorrect.

the predicted next state is:
f = b + x * W

for each component i of the next state, we have:
f_i = b_i + sum over j of x_j * W_ji

so the partials are:
df_i / dW_ji = x_j
...and 0 for all others (i.e. df_i / dW_jk = 0 for all k != i).

More intuitively, it cannot be that e.g. W_11 contributes to f_2; so that partial should be 0. Right now, there are no 0's in the Jacobian because it's something like:
df_i / dW_jk = x_j
for all i,j,k, which is incorrect.

I think the line in update should be more like this:
dXdW = np.tile((np.tile(X, (1, self.D)) * np.eye(self.D)), (self.D, 1))

It probably doesn't matter because we're only using the LDS as a toy example, but would be nice to have it working for sanity checks (and if we're publishing this code)