Comments (9)
Hi @parikshit14
Yes, the best way that I know of to work with multiple repos\packages that depend on each other is to have them each installed to a common virtual environment as editable installs, so you want to install modelspec and MDF like so:
cd modelspec
pip install -e .[dev]
cd ../MDF
pip install -e .[dev]
The install of MDF should skip modelspec since it is already installed. With an editable install, you can simply checkout the development branch of each repo and be working from them. Does this make sense?
I also think creating a helper function that exports a Model instance to all supported formats and imports them back in would be very useful. Checking if the models are equivalent however is a bit tricky. Currently we are defining the == operator on MDF objects to compare simply based on object id. To check for deep equivalence of the nested structure maybe we should implement a method on the model class that checks for deep equivalence. This could be as simple as doing:
def is_equivalent(self, other):
return self.to_dict() == other.to_dict()
If you run into issues where two models aren't equal after serializing pass them to a library like deepdiff to help debug things.
from mdf.
This would be added in modelspec, with save_to_hdf5 etc. in https://github.com/ModECI/modelspec/blob/main/modelspec/utils.py
from mdf.
hi @pgleeson, can I work on this issue?
Is there any particular example for which I can implement this and later I can generalize that. And regarding HDF5, what I got to know about is that it is used to store datasets to store the array, groups which can be made to make the hierarchy and but for attributes it shows in the doc that we cannot manually create it.
from mdf.
Hi @parikshit14 ,
In light of the new changes to the underlying serialization and deserialization code that uses attrs and cattrs, it might be worth exploring using bson or msgpack as binary storage formats for MDF. There are already preconverters built into cattrs for these formats. I think a good place to start might be looking at the code in the modelspec repo to get an understanding of how we are using attrs and cattrs for MDF. Then I would take a look at passing MDF models through the cattrs BSON or msgpack pre converter after they have been serialized to dicts but before you pass them to the msgpack or BSON serializers. I think this would all be less work than an HDF5 format. Some good models to start with are any of the ones in examples/MDF.
from mdf.
Yes, one or both of those two might be a better place to start, it will be much closer to the current structure of MDF when serialised.
I've updated the title of the issue to reflect this.
from mdf.
Should I implement for msgpack as well ?
from mdf.
I think something that might be better would be to move over to the MDF repo now and try to adds some tests and documentation for serializing models to bson. Maybe a test that searches for any json MDF model found in examples and tries to write it to bson and load it back in. Or go through any examples (which are all tested in CI by tests/test_examples.py) and ads test for serializing to bson and loading it back. Maybe we should create a test utility function or a fixture that excepts a model as argument and serializes and deserializes it to all currently supported formats. At the very least we should test it on all the example jsons in examples/MDF.
from mdf.
Hi @davidt0x , is there a way to use the updated version of modelspec(i.e. with the bson changes) which is yet to be released? As an alternative, I tried adding some bson code (as that in modelspec) but I am stuck with to_bson_file()
due to some decoration stuff. I know this is not the best way to declare the same stuff present in modelspec to declare again in MDF but is there any better way?
Also,
I think something that might be better would be to move over to the MDF repo now and try to adds some tests and documentation for serializing models to bson. Maybe a test that searches for any json MDF model found in examples and tries to write it to bson and load it back in. Or go through any examples (which are all tested in CI by tests/test_examples.py) and ads test for serializing to bson and loading it back. Maybe we should create a test utility function or a fixture that excepts a model as argument and serializes and deserializes it to all currently supported formats. At the very least we should test it on all the example jsons in examples/MDF.
from the above-suggested options I am trying to implement the one in which we can export the model to all different exporters like bson, json, yaml and then load it back and assert the values, this is the functionality we want right?
from mdf.
BSON workign well & sufficient for now
from mdf.
Related Issues (20)
- Remove sphinx pin to 3.x? HOT 1
- torch.fx to MDF
- Support for populations/templates HOT 1
- Support for cyclical graphs HOT 3
- onnx 1.14 incompatibility HOT 2
- Py3.10 builds failing due to fastkde HOT 1
- simple_pytorch_to_mdf.py breaking in pytorch 2.0 HOT 2
- Example for testing with Keras->MDF conversion: 3D convolutional neural network HOT 1
- Refactor PyTorch_MDF examples to ease benchmarking HOT 2
- Keras to MDF HOT 1
- Support PyTorch v2.2.0
- Support latest graph_scheduler HOT 5
- ACTR importer condition typo? HOT 1
- Tasks/ Milestones HOT 1
- Missing Link to Functions List in Parameter Section in Documentation HOT 4
- Incorrect GitHub Edit Links in 'modeci_mdf' API Reference Section
- Failure of tests on macos-latest on graphviz setup step
- Allow parameters in Graph to be accessible from any node?
- pypi refusing to make new release as modeci requires verified email HOT 2
- Create a Streamlit based app to run a generic MDF model HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mdf.