How do you save a model after using WOQ? about intel-extension-for-pytorch HOT 2 CLOSED

Hello @eduand-alvarez, you can use the following example to save the quantized mode:

# Save the quantized model
torch.jit.save(torch.jit.script(model), "quantized_model.pt")

In this example, we use PyTorch's JIT (Just-In-Time) compiler to save the quantized model as a TorchScript module. The torch.jit.script function converts the PyTorch model into a TorchScript module, which can be saved to a file using torch.jit.save.
The resulting quantized_model.pt file contains the quantized model weights and architecture, which can be loaded using PyTorch or Hugging Face.

To load the quantized model using PyTorch:

loaded_model = torch.jit.load("quantized_model.pt")

To load the quantized model using Hugging Face:

from transformers import AutoModelForCausalLM
loaded_model = AutoModelForCausalLM.from_pretrained("quantized_model.pt", from_tf=False)

Note that when loading the quantized model using Hugging Face, you need to set from_tf=False to indicate that the model is not a TensorFlow checkpoint

Hope this can be helpful to you.

from intel-extension-for-pytorch.

This is great! Thank you!

from intel-extension-for-pytorch.