[FEATURE] Support for NVidia's A10g about aitemplate HOT 8 CLOSED

I think it will be straightforward to add A10 support, only need to extend the profiling range a little bit. We will make A10 support this week.

from aitemplate.

I enabled A10 (by educated guess how it will output) in this PR: #18

I think to provide better performance on A10, it requires a little bit more extension.

from aitemplate.

+1 on @MatthieuTPHR's request. A10g are widely used for inference and are cheaper wrt to A100s. Can you elaborate a bit more on what extension?

from aitemplate.

I have enabled A10, because we don't have access to A10 I can't guess more on perf optimization for A10. Should be ok.

from aitemplate.

Hello, thanks for this great project.

Following this request it would be amazing to have support for NVidia's latest generation of inference GPUs: the A10g.

They are roughly 2-3x faster than the T4 and very cheap w.r.t A100s.

On another topic, if we wanted to add this support ourselves for this GPU type or any future GPU from NVidia what would be the process ?

Hey @MatthieuTPHR have you tried voltaml stable diffusion library? They support T4 and A10 acceleration and they claim to have the fastest inference speed for now.

from aitemplate.

Hello @harishprabhala looking at their metrics we get a faster inference using Tensor RT.

TensorRT recently added support for Flash Attention here.

With it we get 27 it/s on a A10G compared to the 17 it/s shown in VoltaML's github repo.

On the A100, we get 36 it/s with pytorch and xformers too. I haven't benchmarked the TRT model on the A100 yet though

from aitemplate.

Woah didn't know that. Will look into it. Thanks.

from aitemplate.

Hello @harishprabhala looking at their metrics we get a faster inference using Tensor RT.

TensorRT recently added support for Flash Attention here.

With it we get 27 it/s on a A10G compared to the 17 it/s shown in VoltaML's github repo.

On the A100, we get 36 it/s with pytorch and xformers too. I haven't benchmarked the TRT model on the A100 yet though

Looks like they don't support GPUs with SM<80. So, technically for NVIDIA T4, V100 etc voltaML could be the fastest

from aitemplate.