[Misc]: need "first good issue" about vllm HOT 10 CLOSED

https://github.com/vllm-project/vllm/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22

from vllm.

@HarryWu99 how big of a feature do you want to take on?

from vllm.

how big of a feature do you want to take on?

@robertgshaw2-neuralmagic I don't know yet.😂 I'm still getting familiar with the code. For now, I am interested in "sparse kv cache". Hope to develop a related feature, if my ability allows.

Before that, I'm also happy to deal with some simple bug fixes.

from vllm.

@HarryWu99 cool! A good one to take on eventually would be StreamingLLM in BlockManagerv2

• https://arxiv.org/abs/2309.17453

When I was ramping on the codebase, I found that looking into metrics / monitoring is a decent place to start since you have to touch many pieces

I have a halfway done branch I could use some help pushing over the line

from vllm.

In particular, https://github.com/ronensc/vllm/pull/1/files

Some of this PR I merged in today, but the rest of the new metrics need to be rebased onto main and ensure correctness / testing

If you want to pick it up to get the ball rolling let me know

from vllm.

@robertgshaw2-neuralmagic yes, I'd love to have a try.

from vllm.

Cool - I’m going to pick this back up on Wednesday or so, so please keep me in the loop to your progress

from vllm.

StreamingLLM in BlockManagerv2

@robertgshaw2-neuralmagic Hi, i am interested in implementing this feature, but before the FlashInfer that enable attention kernel to apply rotary embedding in place being merged, are there any decent ways for us to do that?

from vllm.

StreamingLLM in BlockManagerv2

I am interested in it, too.😆

from vllm.

@HarryWu99 we also have some ongoing work for embedding models that could use a hand

from vllm.