Calling ModelSessionService.SaveAsync() During a Queue'd Inference Causes Exception about llamastack HOT 1 CLOSED

Thanks for logging this issue.

Unfortunately if a session has items queued for inference and attemts to load/save are called this could cause and exception as the session is busy.

But this leaves you stuck with no way of knowing when to save, which was an oversight on my part.

Saving a queued session is a bit tricky and I have implemented a fix, however it may not be the final implementation but will allow you to save per message as you are, it adds a SaveOnComplete flag to the 'QueueInferTextAsync' call, setting this to true on an item will save the ModelSDessionState when the queue processes that item, you can provide this on every message or every x up to you

I also expsosed a property InferQueueCount on the ModelSessionService so you can access the queue count if required

from llamastack.