dynamic batching for dali_backend about dali_backend HOT 4 CLOSED

Hello @austingg !
I'm not sure if you are referring to dynamic batching or model concurrency. Anyhow, dali_backend supports both.

In the dynamic batching case, all you need to do is to specify sufficiently large max_batch_size in model configuration and specify the same value in the batch_size argument of DALI pipeline, e.g.

config.pbtxt
------------
backend: dali
max_batch_size: 256


dali_pipeline.py
----------------
@dali.pipeline_def(batch_size=256, num_threads=1, device_id=0)
def pipe():
    images = dali.fn.extenral_source(device="cpu", name="DALI_INPUT_0")
    images = dali.fn.decoders.image(images, device="mixed", output_type=types.RGB)
    images = dali.fn.resize(images, resize_x=299, resize_y=299)
    images = dali.fn.crop_mirror_normalize(images,
                                           dtype=types.FLOAT,
                                           output_layout="HWC",
                                           crop=(299, 299),
                                           mean=[0.485 * 255, 0.456 * 255, 0.406 * 255],
                                           std=[0.229 * 255, 0.224 * 255, 0.225 * 255])
    return images

In case of model concurrency, no special actions are necessary, it should just work. If it doesn't, please let us know.

from dali_backend.

Thanks for your help, @szalpal
I have already used max_batch_size, however other inference backend also specify dynamic batching {} , you mean dali_backend doesn't need this ? I used max_batch_size 256 and instance_group: 10 I have checked get_inference_statisitcs() all dali back batchsize is 1

from dali_backend.

@austingg ,

My apologies. In fact, dynamic batching in dali_backend is not fully supported yet. We expect to handle this in the nearest future, most probably it's going to be included in tritonserver:21.06 release. This feature might be available earlier, but for the main branch build.

I'll keep this issue open until we ship the dynamic batching support

from dali_backend.

looking forward to it. since I used dali_backend as the preprocessing part for an ensemble model, the inference backend can use dynamic batch, the bottleneck is preprocessing.

from dali_backend.