Thread safety for prediction C API TreelitePredictorPredictBatch about treelite HOT 4 CLOSED

@EdisonIsCoding As shown in https://treelite.readthedocs.io/en/latest/treelite-runtime-api.html#treelite.runtime.Predictor.predict, TreelitePredictorPredictBatch function is currently not thread safe. The reason is that it uses a dedicated thread pool to parallelize batch prediction. Consider using TreelitePredictorPredictInst instead.

from treelite.

Hi,

I'm developing a service recently which will allow others to do prediction via my service. And internally, I'm using a treelite predictor and calling into TreelitePredictorPredictBatch to make a prediction. So I'm wandering that whether this API is thread safe, since the request will come from different threads? Thanks for any help in advance!!

from treelite.

@hcho3 Thanks for your timely response! But in the case when the batch only have one row, it looks like TreelitePredictorPredictBatch also works fine, since I ran a stress test with 30 threads and each thread called TreelitePredictorPredictBatch API ~3k times with only one row in the batch (where I should call TreelitePredictorPredictInst in stead), the server did get the right prediction and didn't get into any problem. Is that true?

from treelite.

I'll have to come back to the stability of batch prediction API. I suppose that TreelitePredictorPredictBatch can work in your use case, but I won't provide any guarantees.

from treelite.