failed to compile wkv_cuda about chatrwkv HOT 3 CLOSED

Fix it after upgrading nvcc to newer version.
However, I am not able to run it with 12G VRAM using args.strategy = 'cuda fp16'. It gives me out of memory.
How much memory does it require to run inference of RWKV-4-Pile-7B-Instruct-test4-20230326.pth.?

from chatrwkv.

Read https://pypi.org/project/rwkv/ for strategy guide
You can try 'cuda fp16i8 *20 -> cuda fp16' # first 20 layers cuda fp16i8, then cuda fp16

from chatrwkv.

If I use strategy 'cuda fp16i8 *10 -> cpu fp32',, it runs OK
However, if I enable os.environ["RWKV_CUDA_ON"] = '0' with this strategy 'cuda fp16i8 *10 -> cpu fp32', it gives me errors.
Why is that?

from chatrwkv.