Giter Club home page Giter Club logo

tensorrt_backend's Introduction

License

TensorRT Backend

The Triton backend for TensorRT. You can learn more about Triton backends in the backend repo. Ask questions or report problems on the issues page. This backend is designed to run a serialized TensorRT engine models using the TensorRT C++ API.

Where can I ask general questions about Triton and Triton backends? Be sure to read all the information below as well as the general Triton documentation available in the main server repo. If you don't find your answer there you can ask questions on the main Triton issues page.

Command-line Options

The command-line options configure properties of the TensorRT backend that are then applied to all models that use the backend.

Below is an example of how to specify the backend config and the full list of options.

--backend-config=tensorrt,coalesce-request-input=<boolean>,plugins="/path/plugin1.so;/path2/plugin2.so,version-compatible=true"
  • The coalesce-request-input flag instructs TensorRT to consider the requests' inputs with the same name as one contiguous buffer if their memory addresses align with each other. This option should only be enabled if all requests' input tensors are allocated from the same memory region. The default value is false.

  • The execution-policy flag instructs TensorRT backend to execute the model with different Triton execution policies (see TRITONBACKEND_ExecutionPolicy for detail). Currently the following values are accepted:

    • DEVICE_BLOCKING: corresponds to TRITONBACKEND_EXECUTION_DEVICE_BLOCKING, this option can be set to avoid possible CUDA contention from launching many kernels from multiple threads.
    • BLOCKING: corresponds to TRITONBACKEND_EXECUTION_BLOCKING, this option can be set to overlap the host thread workload between model instances.
  • The plugins flag provides a way to load any custom TensorRT plugins that your models rely on. If you have multiple plugins to load, use a semicolon as the delimiter.

  • The version-compatible flag enables the loading of version-compatible TensorRT models where the version of TensorRT used for building does not matching the engine version used by Triton. You must trust the models loaded in this mode, as version-compatible models include a lean runtime which gets deserialized and executed by Triton. You can find more information in the TensorRT documentation here. The default value is false.

Build the TensorRT Backend

Appropriate version of TensorRT must be installed on the system. Check the support matrix to find the correct version of TensorRT to be installed.

$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install ..
$ make install

The following required Triton repositories will be pulled and used in the build. By default the "main" branch/tag will be used for each repo but the listed CMake argument can be used to override.

  • triton-inference-server/backend: -DTRITON_BACKEND_REPO_TAG=[tag]
  • triton-inference-server/core: -DTRITON_CORE_REPO_TAG=[tag]
  • triton-inference-server/common: -DTRITON_COMMON_REPO_TAG=[tag]

tensorrt_backend's People

Contributors

coderham avatar dyastremsky avatar guanluo avatar krishung5 avatar kthui avatar madhu-nvda avatar mc-nv avatar nv-kmcgill53 avatar oandreeva-nv avatar pskiran1 avatar tabrizian avatar tanmayv25 avatar zhuyujin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.