When I compile TRT8.0 directly, it works well. But when I compile TRT8.0 with this plugin, it failed.

Thanks for the plugin. I converted the model from Pytorch to ONNX format.
How to convert it now to tensorflow format?

When I try to convert, I get an error:
` File "/home/user/anaconda3/envs/pytorch/lib/python3.7/site-packages/onnx_tf/backend.py", line 67, in prepare
super(TensorflowBackend, cls).prepare(model, device, **kwargs)
File "/home/user/anaconda3/envs/pytorch/lib/python3.7/site-packages/onnx/backend/base.py", line 76, in prepare
onnx.checker.check_model(model)
File "/home/user/anaconda3/envs/pytorch/lib/python3.7/site-packages/onnx/checker.py", line 106, in check_model
C.check_model(protobuf_string)
onnx.onnx_cpp2py_export.checker.ValidationError: No Op registered for TRT_PluginV2 with domain_version of 11

==> Context: Bad node spec for node. Name: GridSampler_1441 OpType: TRT_PluginV2`

For some reason, I have to use TensorRT 7.0 in my workspace. It seems like it is feasible to use onnxgraphsurgeon to modify the ONNX graph and then TensorRT will automatically recognize the custom plugin in TensorRT 7.1 or higher version. But is it possible to apply the same changes to TensorRT 7.0?

Does this support 5-D input?

Hi @TrojanXu,

Thanks for your wonderful work, it helps me immensely!

I have implemented it in TRT7 successfully on PC. However, for my platform is the Nvidia Drive Xavier, which is not able to support or upgrade to TRT7, I have to make it work in TRT6.
I have read your code test_plugin_result.py, in my opinion, you replaced GridSampler with TRT_PluginV2 which belongs to TRT7 ops set by using onnx-graphsurgeon of TRT7. Is there possibility to replace GridSampler with an op belongs to TRT6 by using onnx-graphsurgeon of TRT7, and save the modified .onnx file, then using the modified .onnx file directly in TRT6? For I think the TRT7 have to be used just because we have to use onnx-graphsurgeon.

I have compiled your plugin GridSampler in TRT6 without error occured.

Thanks and Regards,
AJ.

Hi! I have already build the grid_sampler in the Tensorrt OSS project and convert my pytorch model to onnx. However it meets an error when I convert onnx to tensorrt. The error is shown below. Could you please tell me how to solve it?

[TensorRT] ERROR: INVALID_ARGUMENT: getPluginCreator could not find plugin ScatterND version 1
In node -1 (importFallbackPluginImporter): UNSUPPORTED_NODE: Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[TensorRT] ERROR: Network must have at least one output
[TensorRT] ERROR: Network validation failed.
Traceback (most recent call last):
File "zed_onnx2trt.py", line 9, in
trt_model = onnx2trt(model)
File "/home/nvidia/.local/lib/python3.6/site-packages/volksdep/converters/onnx2trt.py", line 119, in onnx2trt
trt_model = TRTModel(engine)
File "/home/nvidia/.local/lib/python3.6/site-packages/volksdep/converters/base.py", line 50, in init
self.context = self.engine.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'

Hi,

First, thank you for this repo.

I understand that we need to implement custom layer on the onnx-tensorrt side inside builtin_op_importers.cpp in order tensorrt onnx parser (nvonnxparser) to be able to parse the custom layer. Shouldn't this be enough in order to parse the onnx model file and create a TensorRT inference engine?

I see that you implemented TensorRT plugin and even CUDA implementations for the grid sample op. I thought TensorRT plugins were necessary in order to parse unsupported layers by the uff or caffe parsers (maybe onnx also). But by just implementing custom layer inside builtin_op_importers.cpp and rebuilding onnx-tensorrt, nvonnxparser should be able to parse the model, shouldn't it?

Thanks in advance.

Hi @TrojanXu,

Thanks for your wonderful work.

I have Implemented two custom Plugins for two unsupported Ops. I want to link those custom Plugins for the conversion(while converting from ONNX to TRT Model). Can you please assist me in doing the same? Would be a great help.

Thanks and Regards,
Darshan

When I use the method in this repo to export F.grid_sample in a large model, there is something wrong with the exported graph. The shapes of the inputs of GridSampler nodes are nonetype.

To reproduce

Change the model definition to (Note the +1 in the argument passed to F.grid_sample)

class MyModel(torch.nn.Module):
	def __init__(self):
		super(MyModel,self).__init__()

	def forward(self, input, grid):
		return F.grid_sample(input+1, grid, mode='bilinear', padding_mode='zeros', align_corners=True)

Then export the model, and print the nodeof op GridSampler with the following code.

def check_onnx(onnx_model_file):
	graph = gs.import_onnx(onnx.load(onnx_model_file))
	assert(graph is not None)
	for node in graph.nodes:
		if node.op == 'GridSampler':
			print(node)

And then the console prints

GridSampler_2 (GridSampler)
        Inputs: [Variable (3): (shape=None, dtype=None), Variable (grid): (shape=[1, 1, 128000, 2], dtype=float16)]
        Outputs: [Variable (4): (shape=[1, 17, 1, 128000], dtype=float16)]
Attributes: OrderedDict([('aligncorners', 1), ('interpolationmode', 0), ('paddingmode', 0)])

The desired behavior is that the shape and dtype of the input should not be None.

请问onnx_graphsurgeon是什么包？怎么安装？会提示找不到。

I build the TRT plugin using a makefile and used that in the trtexec to compile my onnx model which has F.grid_sample

While trying to execute it in the engine I get the error

1: [executionContext.cpp::executeInternal::667] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
1: [cudaResources.cpp::~ScopedCudaStream::47] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)

I printed the inputs and there are

mBatch: 1
mInputChannel: 512
mInputHeight: 128
mInputWidth: 128
mGridHeight: 200
mGridWidth: 200
mInterpolationMode: Bilinear
mPaddingMode: Zeros
mAlignCorners: 0
dataType: half

How can I debug this issue?

The grid_sample plugin has inconsistent result with torch.nn.functional.grid_sample.

npy_files:
g_feat.npy: https://drive.google.com/file/d/18YPjH4R8GBew_LZdgziHwze2fMkaYvoP/view?usp=sharing
g_grid.npy: https://drive.google.com/file/d/18tOWjSN-rPUA6odTBBloS7_ERDQ-_a2N/view?usp=sharing

Code to reproduce is as follows,

def torch2onnx():
    import torch
    import numpy as np
    class Debug(torch.nn.Module):
        def __init__(self):
            super(Debug, self).__init__()

        def forward(self, feat, grid):
            return torch.nn.functional.grid_sample(feat, grid, 'bilinear', 'zeros', True)

    net = Debug().cuda()
    net.eval()
    feat = torch.from_numpy(np.load('g_feat.npy')).cuda()
    grid = torch.from_numpy(np.load('g_grid.npy')).cuda()
    with torch.no_grad():
        torch.onnx.export(
            net, (feat, grid), 'gs.onnx', verbose=True,
            input_names=['feat', 'grid'], output_names=['corr'],
            opset_version=16, do_constant_folding=True)

def check_torch_and_trt():
    trt_file = 'gs.trt'
    engine = load_engine(trt_file)

    context = engine.create_execution_context()

    feat = torch.from_numpy(np.load('g_feat.npy')).cuda()
    grid = torch.from_numpy(np.load('g_grid.npy')).cuda()

    out_list = [torch.from_numpy(np.load(key+'.npy')).cuda() for key in ['g_out']]

    out_pred = [torch.empty_like(item) for item in out_list]

    stream = cuda.Stream()
    bindings = [feat.data_ptr(), grid.data_ptr()] + [item.contiguous().data_ptr() for item in out_pred]

    context.execute_async_v2(bindings=bindings, stream_handle=stream.handle)
    #stream.synchronize()

    torch_out = torch.nn.functional.grid_sample(feat, grid, 'bilinear', 'zeros', True)
    print(torch.max(torch.abs(torch_out - out_pred[0])))

Hi, I've added the all the 5 grid sample files to my example project written in TensorRT C++ API. Then, I feed an input and grid that I initialized in my main.cpp to the plugin layer in order to see how it works as follows:

nvinfer1::ITensor* input = network->addInput("INPUT", DataType::kFLOAT, nvinfer1::Dims{4, {1, 1, 4, 4}});
nvinfer1::ITensor* grid = network->addInput("GRID", DataType::kFLOAT, nvinfer1::Dims{4, {1, 8, 8, 2}});
assert(input); assert(grid);

auto interpolation_mode = torch::detail::GridSamplerInterpolation::Bilinear;
auto padding_mode = torch::detail::GridSamplerPadding::Zeros;
auto align_corners = true;

auto creator = getPluginRegistry()->getPluginCreator("GridSampler", "1");
const nvinfer1::PluginFieldCollection* pluginData = creator->getFieldNames();

nvinfer1::PluginFieldCollection mPFC;
std::vector<nvinfer1::PluginField> mPluginAttributes;
mPluginAttributes.emplace_back(nvinfer1::PluginField("interpolation_mode", &interpolation_mode,
                                                    nvinfer1::PluginFieldType::kINT32, 1));
mPluginAttributes.emplace_back(nvinfer1::PluginField("padding_mode", &padding_mode,
                                                    nvinfer1::PluginFieldType::kINT32, 1));
mPluginAttributes.emplace_back(nvinfer1::PluginField("align_corners", &align_corners,
                                                    nvinfer1::PluginFieldType::kINT32, 1));
mPFC.nbFields = mPluginAttributes.size();
mPFC.fields = mPluginAttributes.data();


nvinfer1::IPluginV2 *pluginObj = creator->createPlugin("GridSampler", &mPFC);
nvinfer1::ITensor* inputTensors[] = {input, grid};
auto grid_sample_plugin = network->addPluginV2(inputTensors, 2, *pluginObj);

grid_sample_plugin->getOutput(0)->setName("OUTPUT");
network->markOutput(*grid_sample_plugin->getOutput(0));

The code works but all the values of output are 0. I feed the same input and grid in python, so I know it should not be 0. I am using TensorRT 6.0 and CUDA 10.1. Any idea what I am doing wrong?

Hello, first of all thank you very much for the reference you provided on the custom plugin, but I meet some problems when using this project : I successfully registered this plugin, but when performing inference, the output of engine is a array that all element are zeros，I want to know, what should I do to get the correct output？

why all node.inputs' shape or dtype == None?

Hi @TrojanXu ,

I am trying to convert the Pytorch model to the ONNX model. But whie exporting I am facing an error as: RuntimeError: Exporting the operator grid_sampler to ONNX opset version 12 is not supported.

May I know how to resolve this issue?
As I have seen that you have written Grid sampler TRT plugin, so that can be used while converting from ONNX to TRT right? if yes then how do we have a ONNX model with grid sampler Op(which is not supported in ONNX)?

Thanks and Regards,
Darshan C G

hello, thanks for you proj.
I all process the same as your Readme.md, but I still got UNSUPPORTED_NODE: No importer registered for op: GridSampler when run : LD_PRELOAD=xxx/TensorRT/build/out/libnvinfer_plugin.so.7 python test_plugin_result.py