Classifier-Free Guidance support about aphrodite-engine HOT 4 OPEN

Thanks for the request. As we discussed in private, mirostat support is on the way. After that, we'll focus on a few more deterministic samplers then we can finally move to CFG and grammars.

I have not experimented with exllamav2 kernels yet, but I will assume it'll be straightforward enough to migrate from v1 to v2.

from aphrodite-engine.

At the moment, we've added support for:

• mirostat
• exllamav2 (though not the variable bitrates, GPTQ only)

CFG support is not planned and will likely not happen in the foreseeable future, as it slows down generations.

from aphrodite-engine.

Re-opening this issue so we can keep track of CFG support. After discussing internally, we decided to add it (but not as a high priority addition). We'll likely need to separate CFG requests to their own unique batches, so the throughput cost doesn't affect regular requests.

from aphrodite-engine.

Just saying that as an individual user, having CFG support with 8 bit KV cache is, obviously, really useful. exl2 does this too and it's killer. The net gain for quality feels like it's cheating.
I wonder if there'd be room to further squeeze the second cache - given that it's useful even when you just pass an empty string (or <s>) as the prompt?
I don't suppose the maths of CFG still makes sense with a shorter negative context? ...maybe just kludge it with
"I pinky promise that we found a sink token and not a null reference down there in the context mr transformer don't worry about it just keep up that forward() okay buddy!"

I feel it might not even be a bad thing if the negative guidance was, yknow, low quality? It's gotta stay on topic but
/shrug

Not like I'd know. So I'm greatful that you guys do. Keep it up and thanks for your work.

from aphrodite-engine.