Comments (9)
+1
from optimizer.
Great! I'll support it soon
from optimizer.
I also encounter this problem
from optimizer.
me too
from optimizer.
me too. Is anybody working on in ?
from optimizer.
I am trying converting a model whose size in .pkl is 985 MB, after converting with sklearn-onnx (convert was successful)
Connected to pydev debugger (build 213.7172.26)
[convert_sklearn] parse_sklearn_model
[convert_sklearn] convert_topology
[convert_operators] begin
[convert_operators] iteration 1 - n_vars=0 n_ops=3
[call_converter] call converter for 'SklearnScaler'.
[call_converter] call converter for 'SklearnRandomForestClassifier'.
[call_converter] call converter for 'SklearnZipMap'.
[convert_operators] end iter: 1 - n_vars=6
[convert_operators] iteration 2 - n_vars=6 n_ops=3
[convert_operators] end iter: 2 - n_vars=6
[convert_operators] end.
[_update_domain_version] +opset 0: name='', version=9
[_update_domain_version] +opset 1: name='ai.onnx.ml', version=1
[convert_sklearn] end
Process finished with exit code 1
but then trying to save the onnx model with:
onnx.save(onx, "model.onnx", save_as_external_data=True)
and response
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm 2021.3\plugins\python\helpers\pydev\pydevd.py", line 1483, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm 2021.3\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Workspace/OnnxTest/pklconversion.py", line 126, in <module>
onnx.save(onx, "bonus_model.onnx", save_as_external_data=True)
File "C:\Users\....\AppData\Local\Programs\Python\Python37\lib\site-packages\onnx\__init__.py", line 202, in save_model
s = _serialize(proto)
File "C:\Users\....\AppData\Local\Programs\Python\Python37\lib\site-packages\onnx\__init__.py", line 71, in _serialize
result = proto.SerializeToString()
ValueError: Message onnx.ModelProto exceeds maximum protobuf size of 2GB: 3354372734
I have been finding a fix for this, but I didn't find anything, the question is: Really isn't any solution and you only can't work with onnx with large models?
Thank you very much
from optimizer.
Onnx-ptimizer has supported large model(exceed 2G). You can install the latest version from source build.
from optimizer.
@HSQ79815 could you link to a pull request please? I just tried with the latest onnx weekly build and got the same error when trying to export a large model with save_as_external_data=True
.
from optimizer.
@michaelroyzen you could get more information fromm this pr . In the past, Large model(>2G) is loaded by onnx.load(..., load_external_data=True)
and saved by onnx.save(..., save_as_external_data=True)
, while this functions are implemented byPython
. We implement c++ load and save function that support large model.
from optimizer.
Related Issues (20)
- -1 mean dynamic shape or Static shape? HOT 2
- why using the same cmake but got error HOT 1
- Optimizer reports "Unresolved value references" since v0.3.0 HOT 1
- optimizer support for python 3.11 HOT 5
- CleanUnusedInitializersAndNodeArgs warning still issued after onnxoptimizer run HOT 1
- Support for python3.6 HOT 1
- If model use initializer, fuse_bn_into_conv is broken from v0.3.9 HOT 5
- error on model with sequence types/layers
- local functions missed when optimizing a model
- Question about building with released version ONNX
- optimization passes HOT 3
- Adding onnxoptimizer to conda-forge
- Fold LpNormalization
- Optimizer return an empty model
- How to prohibit constant reuse HOT 2
- 3 concatenations at the same time?
- Failure to install on Python 3.12 HOT 2
- How to add customer fusion pass?
- [Feature request] Integrate Python-only API to ONNX optimizer proposal HOT 2
- Cannot Access Onnx Optimizer Web
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from optimizer.