Comments (3)
Sorry, I'm not sure I understand your question correctly. Here's my attempt at answering it; if this isn't what you're looking for, please try to explain again.
The Generator
and the Constrainer
classes should return the full JSON after execution. Here's an example using Generator
:
from local_llm_function_calling import Generator
import json
functions = [
{
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
"maxLength": 20,
},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
},
}
]
generator = Generator(functions, "gpt2")
function_call = generator.generate("What is the weather like today in Brooklyn?")
function_parameters = json.loads(function_call["parameters"])
function_parameters is then {"location": "{{{{{{{{{{{{{{{{{{{{"}
, which are the AI-generated parameters for the hypothetical get_current_weather
function. Of course, {{{{{{{{{{{{{{{{{{{{
is not a great city name, but that's simply because gpt2
isn't that powerful of a model either.
And here's a Constrainer
example:
from local_llm_function_calling import Constrainer, JsonSchemaConstraint
import json
constrainer = Constrainer("gpt2")
constraint = JsonSchemaConstraint(
{
"type": "object",
"properties": {
"name": {"type": "string", "maxLength": 20},
"age": {"type": "integer"},
},
}
)
generated = constrainer.generate(
"Natural language: I am Jack, I am 20 years old.\nJSON: ",
constraint,
)
validated = constraint.validate(generated)
parameters = json.loads(generated[: validated.end_index])
In this case, the parameters will be {'name': '\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0', 'age': 1}
, which once again is what we described in our schema.
I'll write complete documentation later today, but the summary of how it works is as follows:
- When using a
Generator
, the generator constructs the prompt for the AI. I have yet to play around with the prompts that much so I doubt they're the ideal fit for small models, but it is what it is - you can always create your prompt. It then calls aConstrainer
- A
Constrainer
gives the model theprefix + generated
as the input and gets the probabilities of each token as outputs - Because JSON is a predictable format, I can always say whether specific text adheres to the schema, even if it's incomplete. For example, in the second case, this text breaks the schema:
{
"dreams
because dreams
is not an allowed key in the schema. However, this text:
{
"name
satisfies the JSON schema in this sense, although it isn't yet complete. So, for each token, the Constrainer
checks whether generated + token
breaks the schema (or, instead, it checks whether it satisfies a JsonSchemaConstraint
we provide it with), and appends the highest probability token that doesn't break the schema to the output
- Once the output is complete (the last bracket of the JSON object was closed) or the prompt gets too long for the model, the
Constrainer
returns the generated text - Since the output might include more than just the JSON, we cut it to length using
JsonSchemaConstraint
's output
from local-llm-function-calling.
Why it have problem to correctly extract parameters?
from local-llm-function-calling.
I'm sorry, I don't quite understand your question, can you elaborate on what you mean, preferably by providing a minimal reproducible example and specifying the expected results?
from local-llm-function-calling.
Related Issues (14)
- Problems when trying to use Llama-2 models for function calling HOT 2
- Using CUDA for inference
- Function Calling with LM Studio Server Model HOT 2
- How does this project be compatible with BPE tokenizers? HOT 1
- Lm studio support?
- ERROR: No matching distribution found for local-llm-function-calling HOT 1
- is it possible for us to connect over mail? HOT 1
- Could not install via pip HOT 2
- can this be used for function calling with other opensource models? HOT 7
- Docs example doesn't seem correct HOT 2
- Error at installing HOT 1
- Benchmark data HOT 1
- Function calling Issue HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from local-llm-function-calling.