Comments (2)
The core idea of speculative decoding is that verification is faster than autoregressive generation because the base model can process in batch the entire draft model sample tokens so your intuition in your point 3 is correct
Conceptually this is similar to why prefill is more parallelizable than autoregressive decoding, during the prefill stage you can process the entire context length at once and it would be nice if we could do the same for autoregressive decoding and speculative decoding kinda does that but only for the base model
Back to speculative decoding, how the draft tokens are generated doesn't matter too much it's nowadays popular to generate those draft tokens using a smaller base model but you could guess or have n-gram model, you can train a model etc.. but let's say now you've generated a bunch of sample tokens - how do you verify them?
Well you just base_model(draft_tokens)
and get out the probability that the base_model
assigns to those tokens. Typically once you have those probabilities you can softmax them to get an actual token out but you don't have to do that.
And then if that probability is above some threshold you accept otherwise you reject the new draft tokens and you make the base model generate them.
target_logits = model_forward(
model,
torch.cat([cur_token.view(1), draft_tokens]).view(1, -1),
torch.arange(input_pos, input_pos + speculate_k + 1, device=cur_token.device)
)
target_probs = logits_to_probs(target_logits[0], **sampling_kwargs)
Sidenote: On your question 1 why speculate_k+1
let's say
input_pos = 2
speculate_k = 3
# Generates 2, 3, 4
torch.arange(input_post, input_pos + speculate_k)
# But we actually want 2,3,4,5 hence the +1
I lied a bit above, the way a token gets accepted in speculative decoding isn't a pure threshold though
- You get the probability that base model assigns a token
q
and you compare it to probability of the draft modelq
- If
q > p
it means the base model finds the draft prediction at least as good if not better than probability that the draft model gives it - If
q < p
then you accept the draft token with a a probability ofq / p
- you don't have to do it this way you could outright reject but this is the way they do it in the speculative decoding paper
In the happy path all draft tokens get accepted so now the next model_forward can skip k tokens. And if any token is rejected, you stop at the first rejected location
from gpt-fast.
Related Issues (20)
- What happens to bias during int8 quantization? HOT 3
- Try Tensor Parallel on a server equipped with two V100 linked by NVLINK, but got a performance degradation HOT 8
- batching/dynamic batching HOT 2
- Question about the gennerated code of `WeightOnlyInt8Linear` HOT 5
- AMD RX 7900 XTX Wrong outputs
- Speculative decoding with draft model:TinyLlama-1.1B
- Can't quantize to int4 and can't compile on RTX2080Ti HOT 2
- Int4 perplexity
- index out of range: No transformer config could be loaded HOT 1
- Reducing Latency in Application with Torch Compilation: Initialization and Inference Optimization
- int4/int4-gptq support in Mixtral 8x7B HOT 2
- CUDA error if enabling compile_prefill for quantization model (int8) HOT 7
- RuntimeError: CUDA error: named symbol not found HOT 1
- Size mismatch error occurs when loading models quantized by GPTQ HOT 1
- `eval.py` uses older version of lm_eval HOT 1
- Can GPT-Fast support larger batch sizes HOT 3
- I try to speed up with llava,but this it slower then eager mode,why?
- pass@1 score extremely low using GPT-fast API HOT 2
- Bandwidth achieved for INT8 is much smaller than FP16 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gpt-fast.