Int4(llama-2-chat-7b) converted model generates response in German language about openvino_notebooks HOT 6 CLOSED

Hi @bhardwaj-nakul Since this model is for propose of chat, could you help to try adding a system prompt before the question, and see if the issue happen again?

from openvino_notebooks.

@andreyanufr, can you please take a look and provide an advice on what kind of weight compression configuration should lead to accurate results?

from openvino_notebooks.

@bhardwaj-nakul could you please try to run data aware compression from attached file:
llama_example.txt

from openvino_notebooks.

@bhardwaj-nakul could you please try the example suggested?

from openvino_notebooks.

I tried the provided script. I'm getting below error

The input model to this script is supposed to be FP16 IR model right ? I tried with this itself.

from openvino_notebooks.

@bhardwaj-nakul, this is a known issue and it is already fixed. You should reinstall nncf from Github, like pip install git+https://github.com/openvinotoolkit/nncf.git

from openvino_notebooks.