"Unexpected end of data" when decoding partial Unicode characters with World tokenizer about rwkv.cpp HOT 6 CLOSED

Modify the above file
return self.decodeBytes(tokens).decode('utf-8','ignore')
lost character 녕

@cgisky1980 You can also try the 'replace' mode for errors here, which had better chances for producing all characters properly when using the world models of rwkv.

from rwkv.cpp.

by model rwkv world fp16 is correct

Do you mean that the same code works correctly when using FP16 model?

from rwkv.cpp.

by model rwkv world fp16 is correct

Do you mean that the same code works correctly when using FP16 model?
yes

I mean, models without quantization don't have this problem

from rwkv.cpp.

Then I think this issue actually points at 2 separate problems:

1. quantized model produces less correct text that non-quantized model
2. UnicodeDecodeError error is thrown

For 1, there is no real solution. Quantization reduces quality, it is expected, since information is cut from the model to make it smaller.

2 is an actual bug that can be fixed. I'll put it into my backlog, but anyone can take it.

from rwkv.cpp.

Then I think this issue actually points at 2 separate problems:

1. quantized model produces less correct text that non-quantized model
2. UnicodeDecodeError error is thrown

For 1, there is no real solution. Quantization reduces quality, it is expected, since information is cut from the model to make it smaller.

2 is an actual bug that can be fixed. I'll put it into my backlog, but anyone can take it.

Because this issue is 100% reproducible in models 3B and 7B, I don't think it's a problem of accuracy loss

ps. https:github.com//issues/19 this is the good first issue. LOL

from rwkv.cpp.

Modify the above file
return self.decodeBytes(tokens).decode('utf-8','ignore')
lost character 녕

@cgisky1980 You can also try the 'replace' mode for errors here, which had better chances for producing all characters properly when using the world models of rwkv.

yes，it works

from rwkv.cpp.