Comments (5)
Yes Python runtime is just a simple wrapper of C++ runtime. You can use the generated model with C API directly. This note is helpful: https://github.com/facebookincubator/AITemplate/tree/main/static
And the Model class is showing how to use C API in python, which can take a reference of usage in C++:
https://github.com/facebookincubator/AITemplate/blob/main/python/aitemplate/compiler/model.py
from aitemplate.
Thank you! I try it and I find there is a large difference in using between python runtime and c++ runtime. If you could provided c++ runtime example, we will appreciate it.
Besides, when I want to test multi-instance model, I change the code in "AITemplate/examples/01_resnet-50/benchmark_ait.py", but get error.
The place I changed:
Line 76: mod = Model(os.path.join("./tmp", model_name, "test.so"),2). # add two runtime
Line 89&97: "add num_threads=2, use_unique_stream_per_thread = True"
I meet the error:
[./tmp/resnet50_64/conv2d_bias_relu_373.cu] Got cutlass error: Error Internal at: 154
[16:56:40] ./tmp/resnet50_64/model_interface.cu:158: Error: GraphInstantiate(&graph_exec, graph) API call failed: operation not permitted when stream is capturing at ./tmp/resnet50_64/model-generated.h, line3009
Traceback (most recent call last):
File "benchmark_ait_test.py", line 136, in
main()
File "/home/admin/.local/lib/python3.8/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/home/admin/.local/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/admin/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/admin/.local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "benchmark_ait_test.py", line 132, in main
benchmark("resnet50", batch_size, graph_mode=use_graph)
File "benchmark_ait_test.py", line 89, in benchmark
t, _, __ = mod.benchmark_with_tensors(
File "/home/admin/.local/lib/python3.8/site-packages/aitemplate/compiler/model.py", line 610, in benchmark_with_tensors
mean, std, ait_outputs = self.benchmark(
File "/home/admin/.local/lib/python3.8/site-packages/aitemplate/compiler/model.py", line 566, in benchmark
self.DLL.AITemplateModelContainerBenchmark(
File "/home/admin/.local/lib/python3.8/site-packages/aitemplate/compiler/model.py", line 192, in _wrapped_func
raise RuntimeError(f"Error in function: {method.name}")
RuntimeError: Error in function: AITemplateModelContainerBenchmark
How shell we do to test multi-instance model?
from aitemplate.
cc @mikeiovine
from aitemplate.
This looks like a known issue with graph mode that should be fixed in the next release. For now I recommend setting graph_mode=False
to test this case.
from aitemplate.
Hey @wanghuihhh, so I thought this was a separate issue, but it looks like something else entirely. The problem is that we were starting our stream captures with cudaStreamCaptureModeGlobal
instead of cudaStreamCaptureModeThreadLocal
, causing issues in the multi-threaded tests. I'll have a patch out to fix this particular issue soon.
As for the C++ runtime docs: I'll add a proper example. Thanks for the suggestion!
cc @antinucleon
from aitemplate.
Related Issues (20)
- fx2ait install error -- c10::StorageImpl has no member named mutable_data HOT 1
- AIT seems not able to achieve max along a given dim HOT 9
- complie controlnet error HOT 2
- Stable Diffusion (GLIGEN) Download Error HOT 4
- Building rocm docker image is very slow. Is there a prebuilt docker image? HOT 1
- error during inferencing: Error: Constant embeddings_token_embedding_weight was not set! Set the value with set_constant. HOT 2
- Failed to compile the controlnet: thepowefuldeez/sd21-controlnet-canny HOT 1
- gemm_gen_profiler() takes 3 positional arguments but 4 were given HOT 2
- <class 'src.pipeline_stable_diffusion_ait.StableDiffusionAITPipeline'> is incorrectly implemented. Expected {'feature_extractor', 'scheduler', 'tokenizer', 'text_encoder', 'safety_checker', 'unet', 'vae'} to be defined HOT 4
- `Fatal Python error: Floating point exception` with `run_with_tensors` HOT 3
- gcc: internal compiler error HOT 2
- `Unsupported workload for this conv2d specialization` when using dynamic shape together with permute HOT 6
- Does AIT handle if/else in forward function? HOT 1
- multi-gpu at runtime error HOT 4
- windows platform cannot link _binary_constants_bin_end and _binary_constants_bin_start HOT 5
- Model is successfully compiled, but OOM when loading
- Does Concatenate order matters? HOT 1
- Docker image fails to build due to python dependency issue HOT 2
- Got cutlass error: Error Internal at: 214 HOT 1
- model_interface.cu:231: Error: Constant pretrained_model_patch_embed_proj_weight was not set! Set the value with set_constant. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aitemplate.