Potential encoding issue in addition problem in play_math notebook? about mingpt HOT 3 CLOSED

GPT models use masked attention that ensure their predictions are based only on prior tokens:

Conceptually a GPT model would be evaluated like so:

for i in range(len(x)):
    input = x[:i]
    target = y[i]
    output = model(input)
    loss += loss_func(output, target)

For example, its prediction for the 4th output token is based on the first four input tokens. So it's trying to predict 0 based on just 4, 7, 1, 7.

In practice though the architecture of GPT-like models allow one to compute that entire for loop in a single pass, using masking to ensure that each "column" of the model can only "see" previous tokens.

from mingpt.

@fpgaminer Good catch! Thank you.

It was a miss on my part. I knew that GPT models use masked attention (since they are decoder-only models), but since I was processing this addition problem as a seq2seq formulation, the blindness was on me! So this is a nonissue.

Thanks for clarifying immediately. I would like to keep the issue for a couple of hours, just for illustration to others who might have not recognized this fact, then I will close it.

Thanks
Ravi

from mingpt.

Non-issue, since gpt uses masked attention that always hides token from the future.

from mingpt.