GPU inference? about treelite HOT 4 CLOSED

The RAPIDS Forest Inference Library has a treelite front end for GPU inference. See this blog post: https://medium.com/rapids-ai/rapids-forest-inference-library-prediction-at-100-million-rows-per-second-19558890bc35
The released version (0.10) supports XGBoost and LightGBM models, but the newest nightlies can work with treelite protobufs and sklearn RandomForest models as well. For large-batch inference, it's very very fast, though PCI transfers etc. start to add up if you do single-row inference or very small batches.

(Using treelite as a front-end has been awesome to support all these libraries - many thanks to Philip for his work here.)

from treelite.

@Denisevi4 Interesting suggestion. Only problem is the added latency due to PCIe transfer and kernel launch overhead. For now, we are focusing on getting the best throughput using desktop/server-grade CPUs, and getting a good throughput for small-ish batches (~100) requires low latency.

from treelite.

@Denisevi4 #67 may make GPU inference more practical. See the RapidScorer paper: https://www.kdd.org/kdd2018/accepted-papers/view/rapidscorer-fast-tree-ensemble-evaluation-by-maximizing-compactness-in-data

from treelite.

Please check out RAPIDS cuML, whose GPU-accelerated Forest Inference Library (FIL) integrates with Treelite. Thanks!

from treelite.