Feature: batched inference about fauxpilot HOT 4 OPEN

Now that I've been working with this a little more, I see why this is an issue. Ragged batches are going to be pretty common and we need to do something on our side to do batching. I'm running an experiment that requires generating ~200 tokens and the 6B model takes like 22 seconds on an A10 without batching (~110ms/token sounds too high, not sure what's going on). If batching is enabled and ragged batches exist, then triton processes each request in the batch serially but doesn't return till it's done processing the entire batch. So I see a latency of ~120s because of the fake "batching". I don't see why it does that but that's been my observation.

Here's a log to show that:
https://pastebin.com/36WqBC3F

Btw the current copilot_proxy code calls generate() in the main thread, so there's no concurrency there either. I've fixed that on my local version for now and will send a PR soon.

from fauxpilot.

I have a possibly dumb question -- is there a benefit of doing this? I thought Triton already supports dynamic batching, so why would proxy code need to handle this?

from fauxpilot.

• https://moyix.net/~moyix/batch_codegen_full.py

# Max number of tokens the model can handle
MAX_MODEL_LEN = 2048

FYI, When I actually tested the maximum length of the token using Codegen model, it was possible to 2025 instead of 2048.

from fauxpilot.

I have a possibly dumb question -- is there a benefit of doing this? I thought Triton already supports dynamic batching, so why would proxy code need to handle this?

Oh, you may be right! I should do some perf testing to see if there's any benefit to sending the batch ourselves vs letting Triton batch the inputs. I may have been confused when I read this issue (triton-inference-server/fastertransformer_backend#23) because looking more closely it only applies to ragged batches.

from fauxpilot.