Code difference is getting more between ggml and rwkv.cpp about rwkv.cpp HOT 4 CLOSED

That's a diffucult situation.

On the one hand, I would like to keep ggml up to date, maybe even as a subrepo (so I don't need to manually pull updates). That way we get all new performance improvements and fixes for free.

On the other hand, ggml's Q4_0 and Q4_1 completely break RWKV, and we need Q4_1_O for any hope of quality; and both options "port Q4_1_O again every time ggml updates" and "just delete Q4_1_O" look very bad.

I guess my plan/vision currently is "it works fine now, so we can live without ggml updates at all".

I'll ask in ggml repo what they think about merging Q4_1_O upstream.

from rwkv.cpp.

@yorkzero831 I've forked most recent version of ggml here. The fork contains only one important commit: introduction of Q4_1_O format. I guess it should be usable with Rust now.

from rwkv.cpp.

I started the refactoring in ggml-to-submodule branch.

When merged, this will update ggml in rwkv.cpp to the latest version. After that, I would need to manually pull changes into the submodule to update ggml, but since there is now no new operators and change set is greatly reduced, this would be way easier.

Ideally, I need to pull Q4_1_O format into upstream ggml. That way, I would not even need to support a ggml fork. As an alternative, vanilla ggml quantization formats may become good enough for RWKV, as mentioned in that issue.

from rwkv.cpp.

@saharNooby Really nice, that helps alot

from rwkv.cpp.