wiki proportion finally dominates at the end of the pruning stage about llm-shearing HOT 6 CLOSED

@WangFei-2019 has encountered the same problem. I was helping debug with it but didn't identify the issue yet. Some users have reproduced the experiments without meeting this issue though, so I am also very confused. Will keep you updated once we have more findings about it!

from llm-shearing.

I reran the experiments with my data, and got a curve as this: https://wandb.ai/xmzzzzz/pruning/runs/u7c5y8t5/overview
So I think what you are seeing should mostly be a data problem, though I am not exactly sure why that happens... I will upload my data as soon as I can!

from llm-shearing.

@xiamengzhou Thanks for your reply. I found that the wiki is encoded in UTF-8. Should I decode the wiki to text before feeding it into the tokenizer?

from llm-shearing.

@lippman1125 We have successfully resolved this problem by splitting the Wiki dataset into smaller subfiles and recalculating the reference loss. It is possible that the issue stemmed from inadequate randomness in the data sampling process. Additionally, inconsistencies between the reference loss and the test dataset could also play a role in contributing to this problem.

from llm-shearing.

@WangFei-2019 I split the Wiki dataset into 100 subfiles. Is it enough? Another question: How many sentences did you select for evaluation? The paper said that 500 sentences are enough. Could you share your wiki processing code with me? If it's possible, I would really appreciate it.

from llm-shearing.

@WangFei-2019 Ok. I will have a try. It looks like recalculating the reference loss is a key solution.

from llm-shearing.