[Bug]: After registering the tool for the agent, no matter what information I input, it will always call the tool. about autogen HOT 10 OPEN

I wanted to add some background on LiteLLM when using 'ollama_chat' for the model. LiteLLM will add instruction onto the end of your agent's system message instructing it to only return JSON in a function call format. So this is a big reason why it's difficult to get it to not return a function call.

The text it adds in is:
Produce JSON OUTPUT ONLY! Adhere to this format...

Here's output from litellm (using the --detailed_debug parameter):

[
  {
    "content": "You are a useful AI assistant, and you can call tools to generate random numbers for me. Produce JSON OUTPUT ONLY! Adhere to this format {\"name\": \"function_name\", \"arguments\":{\"argument_name\": \"argument_value\"}} The following functions are available to you:\n{\'type\': \'function\', \'function\': {\'description\': \'get a random number within a given range (inclusive)\', \'name\': \'get_random_number\', \'parameters\': {\'type\': \'object\', \'properties\': {\'low\': {\'type\': \'integer\', \'description\': \'low\'}, \'high\': {\'type\': \'integer\', \'description\': \'high\'}}, \'required\': [\'low\', \'high\']}}}\n",
    "role": "system"
  },
  {
    "content": "hello, give me a number between 0 and 100!",
    "role": "user",
    "name": "User"
  },
  {
    "tool_calls": [
      {
        "id": "call_c62e58c7-ac95-4615-bd07-9e14bbf7a313",
        "function": {
          "arguments": "{\"low\": 0, \"high\": 100}",
          "name": "get_random_number"
        }
      }
    ]
  }
]

I'll keep experimenting with prompts and perhaps trying ollama instead of ollama_chat.

LiteLLM code reference here.

from autogen.

Hey @Noir97 and anyone who's able to run Ollama, please check my draft PR #3056, I've created an Ollama Client class that has tool calling support (preliminary). It will require a lot of testing so if you get a chance to try it out that would be great.

I'll add comments to the PR with results of my tests across different models and test programs.

from autogen.

Hey @Noir97 and anyone who's able to run Ollama, please check my draft PR #3056, I've created an Ollama Client class that has tool calling support (preliminary). It will require a lot of testing so if you get a chance to try it out that would be great.

I'll add comments to the PR with results of my tests across different models and test programs.

I'll test it out in my scenario. Much appreciated for the work and effort.

from autogen.

I'm experiencing the same issue. Any update?

from autogen.

Please see issue #2953.

from autogen.

Besides other fix or improvement in autogen side, may try with adding "...Do not use function unless it is asked" or ""...Do not use tool unless it is asked" in assistant sys message.

from autogen.

Besides other fix or improvement in autogen side, may try with adding "...Do not use function unless it is asked" or ""...Do not use tool unless it is asked" in assistant sys message.

I tried it, but it doesn't seem to work. Even if I say “do not use tools”, it still tries to call it.

from autogen.

@yiriyi ,just did a testing by using gpt-35-turbo, with adding "...Do not use function unless it is asked":

1: Test with allow to use tool

2. Test without use tool

Only using the GPT API seems fine.

from autogen.

I wanted to add some background on LiteLLM when using 'ollama_chat' for the model. LiteLLM will add instruction onto the end of your agent's system message instructing it to only return JSON in a function call format. So this is a big reason why it's difficult to get it to not return a function call.

The text it adds in is: Produce JSON OUTPUT ONLY! Adhere to this format...

Here's output from litellm (using the --detailed_debug parameter):

[
  {
    "content": "You are a useful AI assistant, and you can call tools to generate random numbers for me. Produce JSON OUTPUT ONLY! Adhere to this format {\"name\": \"function_name\", \"arguments\":{\"argument_name\": \"argument_value\"}} The following functions are available to you:\n{\'type\': \'function\', \'function\': {\'description\': \'get a random number within a given range (inclusive)\', \'name\': \'get_random_number\', \'parameters\': {\'type\': \'object\', \'properties\': {\'low\': {\'type\': \'integer\', \'description\': \'low\'}, \'high\': {\'type\': \'integer\', \'description\': \'high\'}}, \'required\': [\'low\', \'high\']}}}\n",
    "role": "system"
  },
  {
    "content": "hello, give me a number between 0 and 100!",
    "role": "user",
    "name": "User"
  },
  {
    "tool_calls": [
      {
        "id": "call_c62e58c7-ac95-4615-bd07-9e14bbf7a313",
        "function": {
          "arguments": "{\"low\": 0, \"high\": 100}",
          "name": "get_random_number"
        }
      }
    ]
  }
]

I'll keep experimenting with prompts and perhaps trying ollama instead of ollama_chat.

LiteLLM code reference here.

Nice find! Have you figured out any possible way we get around this?

from autogen.

Hey @Noir97 - well, yes, somewhat. I'm working on new PRs, which will be for a LiteLLM client and an Ollama client. I'm not sure if I'll submit the LiteLLM one as I've built it to only work with Ollama anyway, so may just go with the Ollama native one.

In these branches I've created a "manual" tool calling approach, whereby the format for the model is no longer set to "JSON" and instead guidance is injected into the prompts and at various stages of tool calling to guide the LLM to return JSON compatible text or just text. Then, I parse the text to see if it is a function call compatible JSON and convert that into a tool call. It is working well, in that I can get it to send through the JSON for the tool call, read the tool call execution and comment on it with normal text.

I feel the current LiteLLM function calling implementation is impossible to get around because it injects the very strict JSON format and also passes a parameter for inference to get a JSON response (though the LLM may still return a non-JSON response).

My working LiteLLM branch is here.

I think it's close enough to being able to be used. If you wanted to test it you could try it out.

You could try this program (change the IP address and model):

import os
import autogen
from typing import Literal
from typing_extensions import Annotated

# THIS TESTS: TOOL CALLING

altmodel_llm_config = {
    "config_list":
    [
        {
            "api_type": "litellm",
            "model": "ollama_chat/mistral:7b-instruct-v0.3-q6_K",
            "api_base": "http://192.168.0.1:11434",
            "cache_seed": None,
            "hide_tools": "if_all_run",
            "tool_calling_mode": "default"
        }
    ]
}

# Create the agent and include examples of the function calling JSON in the prompt
# to help guide the model
chatbot = autogen.AssistantAgent(
    name="chatbot",
    system_message="""For currency exchange tasks,
        only use the functions you have been provided with.
        Output 'TERMINATE' when an answer has been provided.
        Do not include the function name or result in the JSON.
        Example of the return JSON is:
        {
            "parameter_1_name": 100.00,
            "parameter_2_name": "ABC",
            "parameter_3_name": "DEF",
        }.
        Another example of the return JSON is:
        {
            "parameter_1_name": "GHI",
            "parameter_2_name": "ABC",
            "parameter_3_name": "DEF",
            "parameter_4_name": 123.00,
        }. """,

    llm_config=altmodel_llm_config,
)

user_proxy = autogen.UserProxyAgent(
    name="user_proxy",
    is_termination_msg=lambda x: x.get("content", "") and "TERMINATE" in x.get("content", ""),
    human_input_mode="NEVER",
    max_consecutive_auto_reply=1,
)

CurrencySymbol = Literal["USD", "EUR"]

# Define our function that we expect to call
def exchange_rate(base_currency: CurrencySymbol, quote_currency: CurrencySymbol) -> float:
    if base_currency == quote_currency:
        return 1.0
    elif base_currency == "USD" and quote_currency == "EUR":
        return 1 / 1.1
    elif base_currency == "EUR" and quote_currency == "USD":
        return 1.1
    else:
        raise ValueError(f"Unknown currencies {base_currency}, {quote_currency}")

# Register the function with the agent
@user_proxy.register_for_execution()
@chatbot.register_for_llm(description="Currency exchange calculator.")
def currency_calculator(
    base_amount: Annotated[float, "Amount of currency in base_currency"],
    base_currency: Annotated[CurrencySymbol, "Base currency"] = "USD",
    quote_currency: Annotated[CurrencySymbol, "Quote currency"] = "EUR",
) -> str:
    quote_amount = exchange_rate(base_currency, quote_currency) * base_amount
    return f"{format(quote_amount, '.2f')} {quote_currency}"

# start the conversation
res = user_proxy.initiate_chat(
    chatbot,
    message="How much is 123.45 EUR in USD?",
    summary_method="reflection_with_llm",
)

As mentioned, I'll probably move forward with the Ollama one as I'm not sure the LiteLLM provides any benefit as it's built specifically to work with Ollama (rather than be more generalised, which may be the way to go with it). My Ollama one works in exactly the same way except it will be hardcoded to the manual function parsing because Ollama doesn't support tool calling.

from autogen.