LMQL fails to use llama-cpp-python to load models about lmql HOT 3 OPEN

I realize the error with verbosity is a llama-cpp-python bug that has an open issue.

I've updated my example to show the lmql specific issue I am encountering.
I can load the LLM with llama-cpp-python directly and get completions, however, when I try to use lmql I still don't get output, just hanging.

from lmql.

Hi there, I just tried to reproduce this on my workstation, and it seems to all work. Can you make sure to re-install llama.cpp with the correct build flags to enable GPU support?

My test code:

import requests
from pathlib import Path
import lmql

model_file = "/home/luca/repos/lmql/zephyr-7b-beta.Q5_K_M.gguf"

query_string = """
"Q: What is the sentiment of the following review: ```The food was very good.```?\\n"
"A: [SENTIMENT]" where len(TOKENS(SENTIMENT)) < 32
"""

result = lmql.run_sync(
    query_string, 
    model = lmql.model(f"local:llama.cpp:{model_file}", 
        tokenizer = 'HuggingFaceH4/zephyr-7b-beta', 
        verbose=False
    ))

print([result])

Note that I put a token constraint on SENTIMENT, just to make sure we don't have termination issues here. For Colab, please note that sometimes the root of problems with termination is asyncio-related, so maybe try running in a standalone script first.

After re-installing llama.cpp do you still the same issue?

from lmql.

not sure why I got this error, lmql has no attribute 'run_sync'

from lmql.