What is [Prompt len] and [Tokens to Generate]? about gpu_poor HOT 3 CLOSED

You can set it as 1. The results will be almost exactly the same (maybe a 1-2 MB difference at most)

from gpu_poor.

You are right that in fine-tuning/training there is no concept of "prompt Len" & "tokens to generate". Only max_seq_length is required. When you use the GitHub site, the max_seq_length = prompt Len + tokens to Generate.

The prompt Len & context Len concept is only for inference time. For example, if you have a question which is made of 100 words(tokens) and you want to generate an answer of 500 tokens. Here the first 100 tokens are processed at once while the next 500 tokens are processed token by token. Therefore a distinction is needed. The first 100 words are your "prompt Len" & the next 500 words are your "tokens to generate"

from gpu_poor.

Thanks for your reply! So if I want to get the memory result of fine-tuning, I should set "tokens to generate" to 0 right? However that is forbidden (warning that it have to be positive)

from gpu_poor.