nvidia-ai-iot / torch2trt Goto Github PK
View Code? Open in Web Editor NEWAn easy to use PyTorch to TensorRT converter
License: MIT License
An easy to use PyTorch to TensorRT converter
License: MIT License
Hello John,
I am creating a new issue here since I tried the Unet implementation you suggested, and still getting the error that comes up when a layer is nor properly supported in tensorrt. I moved to python3.6 and tensorrt is still 5.0.6.3.
Here is the output of the error:
Traceback (most recent call last):
File "main.py", line 57, in
machine = stateMachine(new_video_FPS, weights)
File "/home/nvidia/Workspace/tfl/tfl-car/aux_func.py", line 534, in init
self.detector = Detector(weights,True)
File "/home/nvidia/Workspace/tfl/tfl-car/aux_func.py", line 443, in init
self.model_trt = torch2trt(self.model, [data], fp16_mode=True)
File "/usr/local/lib/python3.6/dist-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/torch2trt.py", line 252, in torch2trt
outputs = module(*inputs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/nvidia/Workspace/tfl/tfl-car/unet_model.py", line 28, in forward
x = self.up1(x5, x4)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/nvidia/Workspace/tfl/tfl-car/unet_parts.py", line 70, in forward
diffY // 2, diffY - diffY//2))
File "/usr/local/lib/python3.6/dist-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/torch2trt.py", line 97, in wrapper
converter(ctx)
File "/usr/local/lib/python3.6/dist-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/converters/pad.py", line 16, in convert_pad
layer = ctx.network.add_padding(input._trt, pre_padding, post_padding)
AttributeError: 'Tensor' object has no attribute '_trt'
FYI, not sure if this is the model you're using, but i've added layers to support https://github.com/milesial/Pytorch-UNet
Only sigmoid and pad converters were missing.
You can convert the model by
model = UNet(n_channels=3, n_classes=1).cuda().half().eval()
data = torch.zeros((1, 3, WIDTH, HEIGHT)).cuda().half()
model_trt = torch2trt(model, [data], fp16_mode=True)
Speedup on Xavier with my test was about 3X vs plain PyTorch! Let me know if this works for you or you run into any issues.
Best,
John
Originally posted by @jaybdub in #18 (comment)
hello!
When executing the following code I get an initialisation error
modelz = torchvision.models.alexnet(pretrained=True).eval().cuda()
x = torch.ones((1,3,224,224)).cuda()
model_trt = torch2trt(modelz, [x])
traceback:
Traceback (most recent call last):
File "/home/nvidia/Workspace/SCENARIO_3/tfl-car/main.py", line 57, in
machine = stateMachine(new_video_FPS, weights)
File "/home/nvidia/Workspace/SCENARIO_3/tfl-car/aux_func.py", line 584, in init
self.detector = Detector(weights,True)
File "/home/nvidia/Workspace/SCENARIO_3/tfl-car/aux_func.py", line 460, in init
model_trt = torch2trt(modelz, [x])
File "/usr/local/lib/python2.7/dist-packages/torch2trt-0.0.0-py2.7.egg/torch2trt/torch2trt.py", line 244, in torch2trt
builder.create_network() as network, ConversionContext(network) as ctx:
File "/usr/local/lib/python2.7/dist-packages/torch2trt-0.0.0-py2.7.egg/torch2trt/torch2trt.py", line 138, in init
for method, converter in converters.items()
File "/usr/local/lib/python2.7/dist-packages/torch2trt-0.0.0-py2.7.egg/torch2trt/torch2trt.py", line 116, in init
self.method_impl = copy(eval(method))
File "/usr/lib/python2.7/copy.py", line 96, in copy
return _reconstruct(x, rv, 0)
File "/usr/lib/python2.7/copy.py", line 329, in _reconstruct
y = callable(*args)
File "/usr/lib/python2.7/copy_reg.py", line 93, in newobj
return cls.new(cls, *args)
TypeError: instancemethod expected at least 2 arguments, got 0
Any ideas? thanks!
cheers
I create a custom model and register it in the module_test.py to test.
It failed at native cuda. I guess the error is caused by torch.cat operation.
But I can't get providence and no idea how to fix it.
machine: Jetson AGX Xavier
OS: installed from JetPack 4.2.1
python environment: 3.6.8
pytorch version: 1.1.0
The following is the log:
[TensorRT] ERROR: cuda/cudaConvolutionLayer.cpp (99) - Cudnn Error in computeScratchSize: 3
[TensorRT] ERROR: cuda/cudaConvolutionLayer.cpp (99) - Cudnn Error in computeScratchSize: 3
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/media/nvidia/Data/ningl/build/torch2trt/torch2trt/test.py", line 101, in <module>
max_error, fps, fps_trt, ms, ms_trt = run(test)
File "/media/nvidia/Data/ningl/build/torch2trt/torch2trt/test.py", line 30, in run
outputs_trt = module_trt(*inputs_trt)
File "/home/nvidia/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/media/nvidia/Data/ningl/build/torch2trt/torch2trt/torch2trt.py", line 213, in forward
idx = self.engine.get_binding_index(output_name)
AttributeError: 'NoneType' object has no attribute 'get_binding_index'
Attach my model content:
class KitModel(nn.Module):
def __init__(self):
super(KitModel, self).__init__()
self.conv_0 = nn.Conv2d(padding=1, dilation=(1, 1), in_channels=3, out_channels=32, kernel_size=(3, 3), stride=(1, 1), groups=1, bias=True)
self.conv_1 = nn.Conv2d(padding=2, dilation=(2, 2), in_channels=32, out_channels=32, kernel_size=(3, 3), stride=(1, 1), groups=1, bias=True)
self.conv_2 = nn.Conv2d(padding=4, dilation=(4, 4), in_channels=32, out_channels=32, kernel_size=(3, 3), stride=(1, 1), groups=1, bias=True)
self.conv_3 = nn.Conv2d(padding=8, dilation=(8, 8), in_channels=32, out_channels=32, kernel_size=(3, 3), stride=(1, 1), groups=1, bias=True)
self.conv_4 = nn.Conv2d(padding=16, dilation=(16, 16), in_channels=32, out_channels=32, kernel_size=(3, 3), stride=(1, 1), groups=1, bias=True)
self.conv_5 = nn.Conv2d(padding=32, dilation=(32, 32), in_channels=32, out_channels=32, kernel_size=(3, 3), stride=(1, 1), groups=1, bias=True)
self.convf_6 = nn.Conv2d(in_channels=192, out_channels=3, kernel_size=(1, 1), stride=(1, 1), groups=1, bias=True)
def forward(self, x):
conv_0 = self.conv_0(x)
conv_0_activation = F.relu(conv_0)
conv_1 = self.conv_1(conv_0_activation)
conv_1_activation = F.relu(conv_1)
conv_2 = self.conv_2(conv_1_activation)
conv_2_activation = F.relu(conv_2)
conv_3 = self.conv_3(conv_2_activation)
conv_3_activation = F.relu(conv_3)
conv_4 = self.conv_4(conv_3_activation)
conv_4_activation = F.relu(conv_4)
conv_5 = self.conv_5(conv_4_activation)
conv_5_activation = F.relu(conv_5)
concatenate_1 = torch.cat(
[conv_0_activation, conv_1_activation, conv_2_activation,
conv_3_activation, conv_4_activation, conv_5_activation],
1)
convf_6 = self.convf_6(concatenate_1)
convf_6_activation = F.relu(convf_6)
return convf_6_activation
I'm executing the example notebook image_classification/conversion.ipynb and i'm getting:
AttributeError: 'Tensor' object has no attribute '_trt'.
https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/torch2trt/torch2trt.py#L127-L132
We employ TensorRT for accelerating inference in most of case, so I think checking the is_leaf
may prevent converting 'weight' tensor. When we do the inference, we do not use a gradient(sometime for visualization used) and if tensor.requires_grad == True, then the tensor doesn't have is_leaf
so that caused Assert error(if they use detach() it is okay) during the converting.
So, I mean, &it is mainly for accelerating inference whatever the tensor is leaf or not, nobody will care*
I believe that these day a lot of research use self-attention mechanisms which use learnable weight parameter, like input * weight
, then if with this code, it fail to generate trt_
What do you think? Do we have to check is_leaf
, and what for?
Some models have customized layer like corner_pool of cornernet . But how to write customized plugin to support it. The example plugin of interpolate has matched function in PyTorch(torch.nn.functional). It seems some difference as corner_pool layer. Could you give a example of customized plugin like corner_pool ?
Hi, my pytorch version is 1.0.1, TensorRT version is 5.0.2.6
when I use the max pooling layer like this:
nn.MaxPool2d(kernel_size=2, stride=2, ceil_mode=True)
An error occurs:
Tensorrt has no attribute 'PaddingMode'
For Add, generally numpy and pytorch allow to broadcast
For example, a tensor A: shape (1, 3, 224, 224)
with other tensor B: shape(1,)
,
C = A + B
is the tenser C: shape (1, 3, 224, 224)
you can easily reproduce error with
class Add(torch.nn.Module):
def __init__(self):
super(Add, self).__init__()
self.constant = 1
def forward(self, x):
return x + self.constant
Hi
I am using Python2.7, Torch 1.2, cuda 10.0, tensorrt 5.1.4
When trying to load saved tensorrt model by
model_backbone.load_state_dict(torch.load(weights_trt_path))
I got
python: engine.cpp:1104: bool nvinfer1::rt::Engine::deserialize(const void*, std::size_t, nvinfer1::IGpuAllocator&, nvinfer1::IPluginFactory*): Assertion `size >= bsize && "Mismatch between allocated memory size and expected size of serialized engine."' failed.
This model is saved by
from torch2trt import torch2trt
model_trt = torch2trt(model_backbone, [image.unsqueeze(0)])
torch.save(model_trt.state_dict(), 'weights/yolov3-tiny_drone_trt.pth')
With the latest master running Python 2.7, saving TRT model does not work.
# Convert model to TRT
print("Running torch2trt ...")
model = torch2trt(model, [input_t])
print("Saving TRT module ...")
torch.save(model.state_dict(), 'model_trt.pth')
The above works fine with Python 3 installation for torch2trt.
With Python 2.7, the saved model_trt.pth is written out with no warning, but it's only a few hundred bytes.
I am testing with resnet50 and it should be 200 MB+ (and it is with Python 3).
[TensorRT] ERROR: Internal error: could not find any implementation for node (Unnamed Layer* 202) [Deconvolution], try increasing the workspace size with IBuilder::setMaxWorkspaceSize()
[TensorRT] ERROR: ../builder/tacticOptimizer.cpp (1230) - OutOfMemory Error in computeCosts: 0
I have a network (based on ERFNet) and some trained weights for this model. I'm trying to convert it from PyTorch to TensorRT and I do the following:
import torch
from torch2trt import torch2trt
import pdb
from NETWORK_DESIGN import Net
NUM_CLASSES = X
model = Net(NUM_CLASSES)
model_url = torch.load("/path/to/model/model_best.pth")
model.load_state_dict(model_url, strict=False)
model.cuda().half().eval()
x = torch.ones((1, 3, WIDTH, HEIGHT)).cuda().half()
model_trt = torch2trt(model, [x], fp16_mode=True)
pdb.set_trace()
But I get the above error. Is there an unsupported layer in here somewhere? I don't believe there to be any fancy layers in this architecture. I'm happy to share both the network design and weights files. I tested this both on my Laptop (MX150) and a Jetson Xavier (Jetpack 4.2).
The network was trained using PyTorch 1.1.
Hi. Tensorrt 6.0 came out ! It features 3D operators support.
Will we be able to see those added to supported operators in torch2trt as well any time soon ?
> python3 -m torch2trt.test --name=interpolate
Traceback (most recent call last):
File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/site-packages/torch2trt-0.0.0-py3.7.egg/torch2trt/test.py", line 100, in <module>
max_error, fps, fps_trt, ms, ms_trt = run(test)
File "/usr/local/lib/python3.7/site-packages/torch2trt-0.0.0-py3.7.egg/torch2trt/test.py", line 25, in run
module_trt = torch2trt(module, inputs, **self.torch2trt_kwargs)
File "/usr/local/lib/python3.7/site-packages/torch2trt-0.0.0-py3.7.egg/torch2trt/torch2trt.py", line 252, in torch2trt
outputs = module(*inputs)
File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 539, in __call__
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/site-packages/torch2trt-0.0.0-py3.7.egg/torch2trt/converters/interpolate/interpolate.py", line 49, in forward
return F.interpolate(x, self.size, mode=self.mode, align_corners=self.align_corners)
File "/usr/local/lib/python3.7/site-packages/torch2trt-0.0.0-py3.7.egg/torch2trt/torch2trt.py", line 97, in wrapper
converter(ctx)
File "/usr/local/lib/python3.7/site-packages/torch2trt-0.0.0-py3.7.egg/torch2trt/converters/interpolate/interpolate.py", line 34, in convert_interpolate
plugin = get_interpolate_plugin(size=size, mode=mode, align_corners=align_corners)
File "/usr/local/lib/python3.7/site-packages/torch2trt-0.0.0-py3.7.egg/torch2trt/converters/interpolate/interpolate.py", line 11, in get_interpolate_plugin
creator = [c for c in registry.plugin_creator_list if c.name == PLUGIN_NAME and c.plugin_namespace == 'torch2trt'][0]
IndexError: list index out of range
Hi. Can't convert a model loaded with jit trace. However, the same model can be converted when not in trace mode.
model = torch.jit.load(str(checkpoint_fn), map_location="cpu")
model = torch2trt(model, [x])
errors with
File "/usr/local/lib/python3.7/dist-packages/torch2trt/torch2trt.py", line 256, in torch2trt
ctx.mark_outputs(outputs, output_names)
File "/usr/local/lib/python3.7/dist-packages/torch2trt/torch2trt.py", line 171, in mark_outputs
trt_tensor = torch_output._trt
AttributeError: 'Tensor' object has no attribute '_trt'
Hi guys, I noticed that NVIDIA website said TensorRT Python API does not support TX2 and I have found in my own TX2 (jetpack 3.3). When import torch2trt it can't find tensorrt, but I found Jetson benchmark on README.md. So can this project use on TX2? Need I update my TensorRT version to 5?
Update: Update to TensorRT 5 with JetPack 4.2 solve this problem
declare -x JETSON_BOARD="TX1"
declare -x JETSON_CUDA="10.0.326"
declare -x JETSON_CUDA_ARCH_BIN="5.3"
declare -x JETSON_CUDNN="7.5.0.56-1+cuda10.0"
declare -x JETSON_DESCRIPTION="NVIDIA Jetson NANO/TX1"
declare -x JETSON_JETPACK="4.2.1"
declare -x JETSON_L4T="32.2.0"
declare -x JETSON_L4T_RELEASE="32"
declare -x JETSON_L4T_REVISION="2.0"
declare -x JETSON_OPENCV="4.1.0"
declare -x JETSON_OPENCV_CUDA="YES"
declare -x JETSON_SERIAL_NUMBER="14220190355620c08401"
declare -x JETSON_TENSORRT="5.1.6.1-1+cuda10.0"
declare -x JETSON_TYPE="NANO/TX1"
declare -x JETSON_VISIONWORKS="1.6.0.500n"
PYTHON=3.6.8
I am trying to run torch2rt in a jetson nano but trying the code I have the following problems:
model_trt = torch2trt(model, [x])
File "/home/ferran/torch2trt/torch2trt/torch2trt.py", line 263, in torch2trt
ctx.mark_outputs(outputs, output_names)
File "/home/ferran/torch2trt/torch2trt/torch2trt.py", line 173, in mark_outputs
trt_tensor = torch_output._trt
and also I had to modify the code as below since there were problems with enter:
logger = trt.Logger(log_level)
builder = trt.Builder(logger)
network = builder.create_network()
with ConversionContext(network) as ctx:
True
Thanks!
Hi,
I wanted to know what is the difference between serializing a model with torch2trt and PyTorch's TorchScript?
Both seem to have a similar process of serialization. Although I am not sure if TorchScript supports GPUs. Any insight into this will be highly appreciated.
Thanks
Thanks for sharing this amazing work! The torch2trt is good when using single image, but when I expand the number of pictures, it doesn't work. I desire to when does the torch2trt support multiple image input? Thanks for your time!!!
Support int8 calibration
Hi, John.
My model takes a list of 5 Tensors as input.
And when,
model_trt = torch2trt(model, input),
here, input is a list of 5 Tensors.
It gives:
TypeError: forward() takes 2 positional arguments but 6 were given
Hi, it seems that conv3d is not supported. And it seems that onnx2tensorrt does not support it either, how do I get around this issue?
Hi, when I tried to run the example code in README, error occurs like this:
model_trt = torch2trt(net, [x])
File "build/bdist.linux-x86_64/egg/torch2trt/torch2trt.py", line 233, in torch2trt
File "build/bdist.linux-x86_64/egg/torch2trt/torch2trt.py", line 127, in __init__
File "build/bdist.linux-x86_64/egg/torch2trt/torch2trt.py", line 106, in __init__
File "/usr/lib/python2.7/copy.py", line 96, in copy
return _reconstruct(x, rv, 0)
File "/usr/lib/python2.7/copy.py", line 329, in _reconstruct
y = callable(*args)
File "/usr/lib/python2.7/copy_reg.py", line 93, in __newobj__
return cls.__new__(cls, *args)
TypeError: instancemethod expected at least 2 arguments, got 0
The environment of mine is:
OS: Debian 8
Python: 2.7
TRT: 5.1.5.0, which is the latest version
PyTorch: 1.1.0
with this code I can successfully achieve to convert from torch to fp32 and fp16. but I want to convert to int8. How can i do that???
i meet this this problem, when i try the example. i use pytorch0.4.1.
OS: Ubuntu 18.04
GPU: Nvidia 2080Ti
CUDA: 10.1.243
Docker: 19.03.02, build 6a30dfc
Nvidia-docker was installed too
Based on Dockerfile provided in this pull request #68
python -m torch2trt.test
| torchvision.models.alexnet.alexnet | float16 | [(1, 3, 224, 224)] | {'fp16_mode': True} | 6.87E-05 | 1.03e+03 | 1.56e+03 | 0.922 | 0.65 |
/usr/local/lib/python3.6/dist-packages/torchvision/models/squeezenet.py:94: UserWarning: nn.init.kaiming_uniform is now deprecated in favor of nn.init.kaiming_uniform_.
init.kaiming_uniform(m.weight.data)
/usr/local/lib/python3.6/dist-packages/torchvision/models/squeezenet.py:92: UserWarning: nn.init.normal is now deprecated in favor of nn.init.normal_.
init.normal(m.weight.data, mean=0.0, std=0.01)
| torchvision.models.squeezenet.squeezenet1_0 | float16 | [(1, 3, 224, 224)] | {'fp16_mode': True} | 1.95E-03 | 409 | 1.96e+03 | 2.73 | 0.377 |
| torchvision.models.squeezenet.squeezenet1_1 | float16 | [(1, 3, 224, 224)] | {'fp16_mode': True} | 1.95E-03 | 418 | 2.74e+03 | 2.6 | 0.339 |
| torchvision.models.resnet.resnet18 | float16 | [(1, 3, 224, 224)] | {'fp16_mode': True} | 3.91E-03 | 408 | 1.16e+03 | 2.55 | 0.761 |
| torchvision.models.resnet.resnet34 | float16 | [(1, 3, 224, 224)] | {'fp16_mode': True} | 9.38E-02 | 230 | 677 | 4.71 | 1.51 |
| torchvision.models.resnet.resnet50 | float16 | [(1, 3, 224, 224)] | {'fp16_mode': True} | 1.41E-01 | 144 | 765 | 6.57 | 1.36 |
| torchvision.models.resnet.resnet101 | float16 | [(1, 3, 224, 224)] | {'fp16_mode': True} | 0.00E+00 | 80.1 | 371 | 12.3 | 2.78 |
| torchvision.models.resnet.resnet152 | float16 | [(1, 3, 224, 224)] | {'fp16_mode': True} | 0.00E+00 | 55 | 252 | 18 | 3.97 |
/usr/local/lib/python3.6/dist-packages/torchvision/models/densenet.py:212: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
nn.init.kaiming_normal(m.weight.data)
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/dist-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/test.py", line 100, in
max_error, fps, fps_trt, ms, ms_trt = run(test)
File "/usr/local/lib/python3.6/dist-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/test.py", line 25, in run
module_trt = torch2trt(module, inputs, **self.torch2trt_kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/torch2trt.py", line 252, in torch2trt
outputs = module(*inputs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 545, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torchvision/models/densenet.py", line 222, in forward
out = F.avg_pool2d(out, kernel_size=7, stride=1).view(features.size(0), -1)
File "/usr/local/lib/python3.6/dist-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/torch2trt.py", line 97, in wrapper
converter(ctx)
File "/usr/local/lib/python3.6/dist-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/converters/view.py", line 11, in convert_view
layer = ctx.network.add_shuffle(input._trt)
AttributeError: 'Tensor' object has no attribute '_trt'
File "/home/yu/work/videoAnalyze.subject/video_analysis_zh/project_lib/reid_strong_base/modeling/backbones/resnet_cbam/resnet_cbam.py", line 37, in forward
return x.view(x.size(0), -1)
File "/home/yu/anaconda3/envs/py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/torch2trt.py", line 97, in wrapper
converter(ctx)
File "/home/yu/anaconda3/envs/py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/converters/view.py", line 11, in convert_view
layer = ctx.network.add_shuffle(input._trt)
AttributeError: 'Tensor' object has no attribute '_trt'
**I have successfully install with plugins, but I can not import the plugins still, the following is my log. **
[3/3] g++ -shared -o torch2trt/libtorc...obuf-lite -pthread -lpthread -lnvinfer
running install
running bdist_egg
running egg_info
writing torch2trt.egg-info/PKG-INFO
writing dependency_links to torch2trt.egg-info/dependency_links.txt
writing top-level names to torch2trt.egg-info/top_level.txt
reading manifest file 'torch2trt.egg-info/SOURCES.txt'
writing manifest file 'torch2trt.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying torch2trt/init.py -> build/lib/torch2trt
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/libtorch2trt.so -> build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/test.py -> build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/module_test.py -> build/bdist.linux-x86_64/egg/torch2trt
creating build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/pad.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Conv2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/sigmoid.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/BatchNorm2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/add.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/AvgPool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/MaxPool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/ReLU.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/mul.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/cat.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/softmax.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/ConvTranspose2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/AdaptiveAvgPool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/identity.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Linear.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/ReLU6.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/tanh.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/init.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Conv1d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/relu.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/view.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Identity.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/SynchronizedBatchNorm2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/relu6.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/LogSoftmax.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/adaptive_avg_pool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/BatchNorm1d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/div.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/transpose.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/iadd.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
creating build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/interpolate/interpolate.py -> build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/interpolate/init.py -> build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/interpolate/interpolate_pb2.py -> build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/mean.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/init.py -> build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/torch2trt.py -> build/bdist.linux-x86_64/egg/torch2trt
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/test.py to test.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/module_test.py to module_test.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/pad.py to pad.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Conv2d.py to Conv2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/sigmoid.py to sigmoid.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/BatchNorm2d.py to BatchNorm2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/add.py to add.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/AvgPool2d.py to AvgPool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/MaxPool2d.py to MaxPool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/ReLU.py to ReLU.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/mul.py to mul.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/cat.py to cat.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/softmax.py to softmax.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/ConvTranspose2d.py to ConvTranspose2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/AdaptiveAvgPool2d.py to AdaptiveAvgPool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/identity.py to identity.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Linear.py to Linear.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/ReLU6.py to ReLU6.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/tanh.py to tanh.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/init.py to init.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Conv1d.py to Conv1d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/relu.py to relu.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/view.py to view.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Identity.py to Identity.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/SynchronizedBatchNorm2d.py to SynchronizedBatchNorm2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/relu6.py to relu6.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/LogSoftmax.py to LogSoftmax.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/adaptive_avg_pool2d.py to adaptive_avg_pool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/BatchNorm1d.py to BatchNorm1d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/div.py to div.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/transpose.py to transpose.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/iadd.py to iadd.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate/interpolate.py to interpolate.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate/init.py to init.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate/interpolate_pb2.py to interpolate_pb2.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/mean.py to mean.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/init.py to init.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/torch2trt.py to torch2trt.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
torch2trt.pycache.init.cpython-36: module references file
creating 'dist/torch2trt-0.0.0-py3.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing torch2trt-0.0.0-py3.6.egg
removing '/home/yangqh/Envs/torch.1.1/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg' (and everything under it)
creating /home/yangqh/Envs/torch.1.1/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg
Extracting torch2trt-0.0.0-py3.6.egg to /home/yangqh/Envs/torch.1.1/lib/python3.6/site-packages
torch2trt 0.0.0 is already the active version in easy-install.pth
Installed /home/yangqh/Envs/torch.1.1/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg
Processing dependencies for torch2trt==0.0.0
Finished processing dependencies for torch2trt==0.0.0
I have followed #12 and #17, It solves the problem of compile, but I can not import the plugins. So I modify the code of torch2rt/torch2rt/init.py to see what error message is. I comment the "try except",
instead I add the line
load_plugins() PLUGINS_LOADED = True
I rebuilded it with flag -D_GLIBCXX_USE_CXX11_ABI=0, then I use python -m torch2trt.test -name=interpolate to test it again, It occurs the error message
OSError: /home/yangqh/workspace/torch2trt/torch2trt/libtorch2trt.so: undefined symbol: _ZNK6google8protobuf7Message11GetTypeNameEv
If I rebuild it without flag -D_GLIBCXX_USE_CXX11_ABI=0, It occurs error message
/home/yangqh/Envs/torch.1.1/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/libtorch2trt.so: undefined symbol: _ZN3c105ErrorC1ENS_14SourceLocationERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
My environment is
Collecting environment information...
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176
OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.5.1
Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: GPU 0: GeForce GTX 1060 6GB
Nvidia driver version: 390.87
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.1
Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] torch==1.1.0
[pip3] torch2trt==0.0.0
[pip3] torchvision==0.3.0
[conda] Could not collect
Other package:
protobuf-3.9.1/ [build from source]
TensorRT-5.1.5.0/
I don't know what's the problem, do you have any suggestions ? @jaybdub
i had converted the model from pytorch to tensorrt( by torch.save( .pth ) ), can i used it in c++? how?
Has anyone translated the normalize function of pytorch into the torch2trt?
I install torch2trt by --plugins, and I succeed in convert my model to tensorrt model.
And I use torch.save(model_trt.state_dict(), 'test.pth')
then
model_trt = TRTModule()
model_trt.load_state_dict(torch.load('test.pth'))
I met this problem:
[TensorRT] ERROR: getPluginCreator could not find plugin interpolatetorch2trt version 1 namespace torch2trt
[TensorRT] ERROR: Cannot deserialize plugin interpolate
Any suggestions?
Thanks!
model = alexnet(pretrained=True).eval().cuda()
it;s regular pytorch model
how can I convert my alexnet model ?
Hello. I run the sample under the python2.7 but it doesn't run.
I follow the readme and write the following test.
import torch
from torch2trt import torch2trt
from torchvision.models.alexnet import alexnet
# create some regular pytorch model...
model = alexnet(pretrained=True).eval().cuda()
# create example data
x = torch.ones((1, 3, 224, 224)).cuda()
# convert to TensorRT feeding sample data as input
model_trt = torch2trt(model, [x])
y = model(x)
y_trt = model_trt(x)
# check the output against PyTorch
print(torch.max(torch.abs(y - y_trt)))
After I run it. I get the following error:
Traceback (most recent call last):
File "test1.py", line 12, in
model_trt = torch2trt(model, [x])
File "/media/nvidia/Data/ningl/build/torch2trt/torch2trt/torch2trt.py", line 244, in torch2trt
builder.create_network() as network, ConversionContext(network) as ctx:
File "/media/nvidia/Data/ningl/build/torch2trt/torch2trt/torch2trt.py", line 138, in init
for method, converter in converters.items()
File "/media/nvidia/Data/ningl/build/torch2trt/torch2trt/torch2trt.py", line 116, in init
self.method_impl = copy(eval(method))
File "/usr/lib/python2.7/copy.py", line 96, in copy
return _reconstruct(x, rv, 0)
File "/usr/lib/python2.7/copy.py", line 329, in _reconstruct
y = callable(*args)
File "/usr/lib/python2.7/copy_reg.py", line 93, in newobj
return cls.new(cls, *args)
TypeError: instancemethod expected at least 2 arguments, got 0
Hi,
I'm using Python 2.7, TensorRT 5.1.5.0 and Pytorch 1.2.0 and I want to convert my model with torch2trt, but I got error in BatchNorm1d
, here is the output.
[TensorRT] ERROR: Parameter check failed at: ../builder/Network.cpp::addScale::175, condition: shift.count == 0 || shift.count == weightCount
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/torch2trt-0.0.0-py2.7.egg/torch2trt/torch2trt.py", line 252, in torch2trt
outputs = module(*inputs)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 547, in __call__
result = self.forward(*input, **kwargs)
File "Model/models.py", line 51, in forward
return self.layers(inputs[0])
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 547, in __call__
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 547, in __call__
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/torch2trt-0.0.0-py2.7.egg/torch2trt/torch2trt.py", line 97, in wrapper
converter(ctx)
File "/usr/local/lib/python2.7/dist-packages/torch2trt-0.0.0-py2.7.egg/torch2trt/converters/BatchNorm1d.py", line 22, in convert_BatchNorm2d
layer = ctx.network.add_shuffle(layer.get_output(0))
AttributeError: 'NoneType' object has no attribute 'get_output'
And here is my model.
>>> print(model)
Baseline(
(layers): Sequential(
(0): Dropout(p=0.25, inplace=False)
(1): Linear(in_features=2000, out_features=256, bias=True)
(2): ReLU()
(3): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(4): Dropout(p=0.25, inplace=False)
(5): Linear(in_features=256, out_features=128, bias=True)
(6): ReLU()
(7): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(8): Dropout(p=0.25, inplace=False)
(9): Linear(in_features=128, out_features=2, bias=True)
)
)
>>> x = torch.zeros([1, 2000]).to('cuda:1')
>>> print(model(x))
tensor([[0., 0.]], device='cuda:1', grad_fn=<AddmmBackward>)
>>> model_trt = torch2trt(model, [x]) # got error
Hi,
I serialized the model, and then loaded the model in C++ API following #16
But when I sent a tensor with all ones into the models (pytorch model and serialized model in c++), their outputs are different.
when I sent a tensor with all zeros, the outputs are the same.
My TensorRT version is 5.0.2.6
PyTorch version is 1.1.0
I get error AttributeError: 'Tensor' object has no attribute '_trt'
when I meet MaxPool2d
/usr/local/lib/python3.6/dist-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/torch2trt.py in wrapper(*args, **kwargs)
95
96 #print('%s : %s' % (method.qualname, converter.name))
---> 97 converter(ctx)
98
99 # convert to None so conversion will fail for unsupported layers
/usr/local/lib/python3.6/dist-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/converters/MaxPool2d.py in convert_MaxPool2d(ctx)
21
22 layer = ctx.network.add_pooling(
---> 23 input=input._trt, type=trt.PoolingType.MAX, window_size=kernel_size)
24 layer.stride = stride
25 layer.padding = padding
AttributeError: 'Tensor' object has no attribute '_trt'
I receive this error whenever I try to run my model (Resnet IR_50 or Resnet IR_152) converted to torchrt on an input.
Code:
with torch.no_grad():
input = torch.randn(1,3,112,112)
input = input.to(device)
jit = torch.jit.trace(backbone, input)
trt = torch2trt(backbone, [input])
for i in range(10):
tic = time.time()
x = trt(input)
torch.cuda.synchronize()
print('trt forward time: {:.4f}'.format(time.time() - tic))
tic = time.time()
y = backbone(input)
torch.cuda.synchronize()
print('backbone forward time: {:.4f}'.format(time.time() - tic))
tic = time.time()
z = jit(input)
torch.cuda.synchronize()
print('jit forward time: {:.4f}'.format(time.time() - tic))
print('TRT Error: ', torch.max(torch.abs(y - x)))
print('JIT Error: ', torch.max(torch.abs(y - z)))
Output:
... x 10
[TensorRT] ERROR: INVALID_ARGUMENT: Can not find binding of given name
trt forward time: 0.0031
backbone forward time: 0.0067
jit forward time: 0.0065
TRT Error: tensor(1.6327, device='cuda:0')
JIT Error: tensor(6.1989e-06, device='cuda:0')
Sidenote: I had to add synchronize in because torch.cudnn.benchmark = True causes a crash in torch2trt
x1 = torch.ones((1, 3, 224, 224)).cuda()
model_trt = torch2trt(model, [x])
y = model(x1)
y_trt = model_trt(x1)
print(torch.max(torch.abs(y - y_trt)))
This code returns 0, for every element.
I expected the same thing from
x2 = torch.ones((1, 3, 224, 224)).cuda() * 20
y = model(x2)
y_trt = model_trt(x2)
print(torch.max(torch.abs(y - y_trt)))
but this code returns big differences. if I increase distance between x1 and x2, difference increase more
Hello, thank you for sharing this amazing project!!! With the help of torch2trt, the running speed of model in test phrase is very fast!!! However, this miracle seems to happen in a single gpu card. Therefore, I desire to know if torch2trt can support multi gpu cards like "torch.nn.DataParallel". Thanks for you time
This happened when I tried using the code in the notebooks folder into a Python script
Traceback (most recent call last):
File "conversion.py", line 5, in <module>
from torch2trt import torch2trt
File "/mnt/886E59876E596F46/PycharmProjects/torch2trt/torch2trt/__init__.py", line 1, in <module>
from .torch2trt import *
File "/mnt/886E59876E596F46/PycharmProjects/torch2trt/torch2trt/torch2trt.py", line 237, in <module>
def torch2trt(module, inputs, input_names=None, output_names=None, log_level=trt.Logger.ERROR, max_batch_size=1,
AttributeError: module 'tensorrt' has no attribute 'Logger'
'TRTModule' object has no attribute '_register_state_dict_hook'
my pytorch version is 0.4.1
tensorrt5.0.2.6
This is more of a question rather than an issue, but does torch2trt support multiple inputs as in a list/dict of tensors?
I've been eyeing torch2trt as an alternative to torch.onnx.export because the latter needs conversion to onnx & also only accepts one dummy_input.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.