Error when `top_logprobs` value is `-inf` about aphrodite-engine HOT 4 CLOSED

In what scenario are you getting -inf as the top logprob? That's really weird

from aphrodite-engine.

I was sending logit_bias and thought it might be the issue, but I removed it and the error persists

{'301': 100.0, '528': 100.0, '626': 100.0, '766': 100.0, '885': 100.0, '1153': 100.0, '1424': 100.0, '1472': 100.0, '1999': 100.0, '4966': 100.0, '9613': 100.0, '9796': 100.0, '11158': 100.0, '12327': 100.0, '12758': 100.0, '14610': 100.0, '15377': 100.0, '18782': 100.0, '19253': 100.0, '21104': 100.0, '21620': 100.0, '22314': 100.0, '23407': 100.0, '23451': 100.0, '24173': 100.0, '25945': 100.0, '26230': 100.0, '27719': 100.0}

These are my SampingParams:

SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.17, temperature=1.31, top_p=0.14, top_k=49, top_a=0.52, min_p=0.0, tfs=1.0, eta_cutoff=10.42, epsilon_cutoff=1.49, typical_p=1.0, mirostat_mode=0, mirostat_tau=5.0, mirostat_eta=0.1, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=['\n###', '</s>', '<|', '\n#', '\n\n\n'], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1, custom_token_bans=[], logprobs=10, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True)

I'm using TheBloke/MythoMax-L2-Kimiko-v2-13B-GPTQ for this case
I can debug further until I find why this happens.

from aphrodite-engine.

i get this with miku and tsukasa, mikus hotfix fixed for me

from aphrodite-engine.

Sorry I forgot to get back to you on this. Yes, AFAIK, JSON doesn't support -inf as a numeric value. If you can open a PR to fix that, it'd be great.

And yes, -inf value is expected; happens when the probs for a token is effectively zero. The log of zero is -inf. Your workaround of setting -inf to -1000 should be fine.

from aphrodite-engine.