Integrate `safetensors` for model serialization about zenml HOT 4 OPEN

Correct.

from zenml.

Hi @strickvl I want to work on this task.

By reading your detailed description I found out that safetensors could be used for Huggingface, PyTorch, PyTorch Lightning and Tensorflow integrations. And we could change the current method to use safetensors to store the model. Please correct me if I'm wrong here.

Here I could add a new materialize named for eg. SafetensorsMaterializer similar to CloudPickleMaterializer.

But I'm struggling with figuring out how can we show users both the option of pickle and safetensors, I mean where can we make this change? As I'm new to this repo, please can you guide me a little on this? Thanks :)

from zenml.

Hi @Dev-Khant good question!

I think what would be the best first place to start would be simply to add new materializers that use safetensors. Then we can allow users to specify them as a custom materializer for their chosen outputs. (See here for more details on that).

We can keep the new materializers as part of the standard library, but they just wouldn't be the default. (The alternative would be to have a config option on the materializer itself, but that's a big / complicated feature to implement and I think we shouldn't start there).

So, don't change the existing materializers but add new ones that use safetensors and update the docs so that people know how to use these parallel options. Hope that makes sense!

from zenml.

@strickvl Totally Understood. As you said we will have parallel options for materialized, so correct if me I am wrong, we will have let's say two HFPTModelMaterializer one would be with the current approach and another one with safetensors.

from zenml.