About the parameter extraction? about local-llm-function-calling HOT 3 CLOSED

Sorry, I'm not sure I understand your question correctly. Here's my attempt at answering it; if this isn't what you're looking for, please try to explain again.

The Generator and the Constrainer classes should return the full JSON after execution. Here's an example using Generator:

from local_llm_function_calling import Generator
import json

functions = [
    {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA",
                    "maxLength": 20,
                },
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        },
    }
]
generator = Generator(functions, "gpt2")

function_call = generator.generate("What is the weather like today in Brooklyn?")
function_parameters = json.loads(function_call["parameters"])

function_parameters is then {"location": "{{{{{{{{{{{{{{{{{{{{"}, which are the AI-generated parameters for the hypothetical get_current_weather function. Of course, {{{{{{{{{{{{{{{{{{{{ is not a great city name, but that's simply because gpt2 isn't that powerful of a model either.

And here's a Constrainer example:

from local_llm_function_calling import Constrainer, JsonSchemaConstraint
import json


constrainer = Constrainer("gpt2")
constraint = JsonSchemaConstraint(
    {
        "type": "object",
        "properties": {
            "name": {"type": "string", "maxLength": 20},
            "age": {"type": "integer"},
        },
    }
)

generated = constrainer.generate(
    "Natural language: I am Jack, I am 20 years old.\nJSON: ",
    constraint,
)
validated = constraint.validate(generated)
parameters = json.loads(generated[: validated.end_index])

In this case, the parameters will be {'name': '\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0', 'age': 1} , which once again is what we described in our schema.

I'll write complete documentation later today, but the summary of how it works is as follows:

• When using a Generator, the generator constructs the prompt for the AI. I have yet to play around with the prompts that much so I doubt they're the ideal fit for small models, but it is what it is - you can always create your prompt. It then calls a Constrainer
• A Constrainer gives the model the prefix + generated as the input and gets the probabilities of each token as outputs
• Because JSON is a predictable format, I can always say whether specific text adheres to the schema, even if it's incomplete. For example, in the second case, this text breaks the schema:

{
    "dreams

because dreams is not an allowed key in the schema. However, this text:

{
    "name

satisfies the JSON schema in this sense, although it isn't yet complete. So, for each token, the Constrainer checks whether generated + token breaks the schema (or, instead, it checks whether it satisfies a JsonSchemaConstraint we provide it with), and appends the highest probability token that doesn't break the schema to the output

• Once the output is complete (the last bracket of the JSON object was closed) or the prompt gets too long for the model, the Constrainer returns the generated text
• Since the output might include more than just the JSON, we cut it to length using JsonSchemaConstraint's output

from local-llm-function-calling.

Why it have problem to correctly extract parameters?

from local-llm-function-calling.

I'm sorry, I don't quite understand your question, can you elaborate on what you mean, preferably by providing a minimal reproducible example and specifying the expected results?

from local-llm-function-calling.