Allow serialisation of MDF in a binary format (HDF5, bson or msgpack) about mdf HOT 9 CLOSED

Hi @parikshit14

Yes, the best way that I know of to work with multiple repos\packages that depend on each other is to have them each installed to a common virtual environment as editable installs, so you want to install modelspec and MDF like so:

cd modelspec
pip install -e .[dev]
cd ../MDF
pip install -e .[dev]

The install of MDF should skip modelspec since it is already installed. With an editable install, you can simply checkout the development branch of each repo and be working from them. Does this make sense?

I also think creating a helper function that exports a Model instance to all supported formats and imports them back in would be very useful. Checking if the models are equivalent however is a bit tricky. Currently we are defining the == operator on MDF objects to compare simply based on object id. To check for deep equivalence of the nested structure maybe we should implement a method on the model class that checks for deep equivalence. This could be as simple as doing:

def is_equivalent(self, other):
     return self.to_dict() == other.to_dict()

If you run into issues where two models aren't equal after serializing pass them to a library like deepdiff to help debug things.

from mdf.

This would be added in modelspec, with save_to_hdf5 etc. in https://github.com/ModECI/modelspec/blob/main/modelspec/utils.py

from mdf.

hi @pgleeson, can I work on this issue?
Is there any particular example for which I can implement this and later I can generalize that. And regarding HDF5, what I got to know about is that it is used to store datasets to store the array, groups which can be made to make the hierarchy and but for attributes it shows in the doc that we cannot manually create it.

from mdf.

Hi @parikshit14 ,

In light of the new changes to the underlying serialization and deserialization code that uses attrs and cattrs, it might be worth exploring using bson or msgpack as binary storage formats for MDF. There are already preconverters built into cattrs for these formats. I think a good place to start might be looking at the code in the modelspec repo to get an understanding of how we are using attrs and cattrs for MDF. Then I would take a look at passing MDF models through the cattrs BSON or msgpack pre converter after they have been serialized to dicts but before you pass them to the msgpack or BSON serializers. I think this would all be less work than an HDF5 format. Some good models to start with are any of the ones in examples/MDF.

from mdf.

Yes, one or both of those two might be a better place to start, it will be much closer to the current structure of MDF when serialised.

I've updated the title of the issue to reflect this.

from mdf.

Should I implement for msgpack as well ?

from mdf.

I think something that might be better would be to move over to the MDF repo now and try to adds some tests and documentation for serializing models to bson. Maybe a test that searches for any json MDF model found in examples and tries to write it to bson and load it back in. Or go through any examples (which are all tested in CI by tests/test_examples.py) and ads test for serializing to bson and loading it back. Maybe we should create a test utility function or a fixture that excepts a model as argument and serializes and deserializes it to all currently supported formats. At the very least we should test it on all the example jsons in examples/MDF.

from mdf.

Hi @davidt0x , is there a way to use the updated version of modelspec(i.e. with the bson changes) which is yet to be released? As an alternative, I tried adding some bson code (as that in modelspec) but I am stuck with to_bson_file() due to some decoration stuff. I know this is not the best way to declare the same stuff present in modelspec to declare again in MDF but is there any better way?

Also,

I think something that might be better would be to move over to the MDF repo now and try to adds some tests and documentation for serializing models to bson. Maybe a test that searches for any json MDF model found in examples and tries to write it to bson and load it back in. Or go through any examples (which are all tested in CI by tests/test_examples.py) and ads test for serializing to bson and loading it back. Maybe we should create a test utility function or a fixture that excepts a model as argument and serializes and deserializes it to all currently supported formats. At the very least we should test it on all the example jsons in examples/MDF.

from the above-suggested options I am trying to implement the one in which we can export the model to all different exporters like bson, json, yaml and then load it back and assert the values, this is the functionality we want right?

from mdf.

BSON workign well & sufficient for now

from mdf.