Some weights of LlamaMoEForCausalLM were not initialized about llama-moe HOT 5 CLOSED

from llama-moe.

Hi there, thanks for your interest in this project~

inv_freq will be re-initialized from scratch each time when we initialize a model. This is NOT expected if you have fine-tuned or changed the inv_freq. In that case, this parameter should be in part of the dumpped model weights.

from llama-moe.

Hi there, thanks for your interest in this project~

inv_freq will be re-initialized from scratch each time when we initialize a model. This is NOT expected if you have fine-tuned or changed the inv_freq. In that case, this parameter should be in part of the dumpped model weights.

So, this error "" Some weights of LlamaMoEForCausalLM were not initialized" "has no effect?

from llama-moe.

There are no effects if you don't change this part of weights and could accept its re-initialization. I checked the model weights in Qwen/Qwen1.5-4B but found no inv_freq. So the re-initialization is a expected behaviour.

from llama-moe.

There are no effects if you don't change this part of weights and could accept its re-initialization. I checked the model weights in Qwen/Qwen1.5-4B but found no inv_freq. So the re-initialization is a expected behaviour.

Thank you very much. I see.

from llama-moe.