`dvclive` integration? about mlem HOT 3 OPEN

Is saving the only action that dvclive wants to do with ML models?

As of today, yes. But maybe @dberenbaum can give additional insights from https://www.notion.so/iterative/Lightweight-Experiments-cec7b7c6e3d2451490491d22dfb4e63f . We are moving towards taking care of saving the model in all ML Framework integrations. Currently we are using the "native" ML Framework mechanism for saving the model (i.e. keras.model.save).

It's definitely worth discussing this in more details. I'm not sure how to integrate this with dvclive properly and what parts do we need to implement twice in both libs.

For me, it feels like DVCLive main value is to reduce the amount of changes users need to benefit from DVC features, without having to care about the specifics.

It seems like a good direction to discuss if a similar sinergy with MLEM could exist; use DVCLive to make MLEM features more easily accesible from a DVC pipeline.

The way I currently see it is that we could add mlem.save calls as part of each DVCLive integration. I'm not sure if the fact that DVCLive would know the ML Framework beforehand could be used to make assumptions on this mlem.save calls.

As a side note, although it might be too early to discuss, it seems that the work being done in iterative/dvc#6332 could be another potential place for synergy between DVC checkpoints and mlem.publish (although I don't know if MLEM considers an intermetiade checkpoint a model or it's more focused on the "post train" model).

from mlem.

Hi, @daavoo! Thanks for submitting this. Sorry for the long delay, wasn't subscribed to repo issues.
FIrst of all, feel free to tag me and @mike0sv on any question regarding MLEM, right now we are the two working on it.

Regarding the question itself: indeed, one of fundamental functions of MLEM is to save models, whether locally or via committing them to the remote repo (check "Save model" header here https://www.notion.so/iterative/Creating-Model-Registry-1b4383745de349a48bd2c08fffcfda63#9b3137ec520f4902a646635b01795a4d).

Right now MLEM has a powerful mechanism to identify what model is supplied to mlem.save(model, path) to save it appropriately (e.g., is it sklearn model, lightgbm, torch, or something else).

Is saving the only action that dvclive wants to do with ML models?

It's definitely worth discussing this in more details. I'm not sure how to integrate this with dvclive properly and what parts do we need to implement twice in both libs.

from mlem.

I'm not sure if the fact that DVCLive would know the ML Framework beforehand could be used to make assumptions on this mlem.save calls.

In general, there should be no need to know ML Framework beforehand because MLEM finds it out under the hood inspecting the obj you pass to mlem.save(obj, path).

Also, we've been busy with finishing prototype and product design tasks for MLEM, so this was on hold for us for some time. We can get back to this in the second half of September, when the model saving functionality in MLEM will be ready for the first closed release. Or if you feel that we need to start discussing this earlier for some reason, please let me know.

from mlem.