TGI crashes with complex json schemas provided as grammar without any information (on debug/trace level) about text-generation-inference HOT 3 OPEN

Hey @o1iv3r thanks for sharing! I'll try to reproduce soon and share an update here.

from text-generation-inference.

FYI, I think it might be a problem in the outlines library which also doesn't work for me with a large number of fields.

from text-generation-inference.

Hi!
The LLM has to worry about generating the JSON as well as the fields in the schema, I think that's the issue. Grammar works 99% of the time really well with smaller schemas. I have to admit I've never seen a schema so long, but the use-case is absolutely something that should work effectively.
I've been doing some reading around schema based generation and I came across this article from Lamini here... it looks like they present to the LLM the pre-filled JSON, this saves on compute plus all the LLM has to do is generate the field contents. The schema parsing would never fail this way.
@drbh I'm not totally sure on the implementation currently in TGI but I'm assuming the LLM is also generating the JSON right now. Is there scope to implement something like this going forwards? I can see great benefit in this if so :)
Thanks.

from text-generation-inference.