Question about Training about mlora HOT 6 CLOSED

• The repo show how to test the throughput.
• This project is now just used for the train, the inference's implementation will be added later(the main branch's inference just for test).

from mlora.

Thanks for replying to me.
Yes, I understand the patch.
I just got confused by the feed-forward part in the tuning session,
aren't we supposed to do auto-regression in the feed-forward parts?

from mlora.

Do you mean why there is only use the output to train, not like the inference produces token by token and calc the MSELoss?

from mlora.

Yes,
I was thinking aren't we supposed to first generate a series of outputs and then use the output tokens to compare with the ground truth label?
Like: "ML is Fun" as input, Out put is "ML is Fun topic to work on" which add 4 words(maybe 12 tokens)
Then the ground truth is "ML is Fun topic indeed"
Then we compare these two.

from mlora.

I think sequence classification/regression does this, if we just train for chat, cross-entropy loss maybe better.
If your input is "ML is Fun topic indeed.", the output will contain all sub-sequence predictions, "ML xx", "ML is xx", "ML is Fun xx", "ML is Fun topic xx". Then we calc the cross-entropy loss.
But we can add the classification/regression features later if needed.

from mlora.

since no comments, i would like to close this one.

from mlora.