stride value caused skipping one word about llms-from-scratch HOT 2 CLOSED

That's a good point, thanks for bringing that up. Just pasting the relevant passage here for context:

Note that we increase the stride to 5, which is the max length + 1. This is to utilize the data set fully (we don't skip a single word) but also avoid any overlap between the batches, since more overlap could lead to increased overfitting. For instance, if we set the stride equal to the max length, the target ID for the last input token ID in each row would become the first input token ID in the next row.

And the tokens as they appear in tokenizer.encode(raw_text) are as follows:

[40, 367, 2885, 1464, 1807, 3619, 402, 271, 10899, 2138, 257, 7026, 15632, 438, 2016, ...]

max_len=4, stride=5:

In the book I (and code) I used the following setting:

batch 1
x: [40,   367,   2885,  1464]
y: [367,  2885,  1464,  1807]

batch 2
x: [ 3619,   402,   271, 10899]
y: [  402,   271, 10899,  2138],

max_len=4, stride=4:

The alternative, without skipping a token in x, is:

batch 1
x: [  40,   367,   2885,  1464]
y: [  367,  2885,  1464,  1807]

batch 2
x: [ 1807,  3619,   402,   271]
y: [ 3619,   402,   271, 10899]

In this case, the token 1807 is "repeated." I.e. it appears as a target in y (batch 1) and input in x (batch 2). It's not repeated in the input, but it gets used 2x in a way. But thinking about this more, I think this max_len=4, stride=4 should probably be preferred.

from llms-from-scratch.

Just updated it in the code and manuscript. Thanks again.

from llms-from-scratch.