opengenerativeai / llm-colosseum Goto Github PK

View Code? Open in Web Editor NEW

957.0 16.0 110.0 45.4 MB

Benchmark LLMs by fighting in Street Fighter 3! The new way to evaluate the quality of an LLM

Home Page: https://huggingface.co/spaces/junior-labs/llm-colosseum

License: MIT License

Python 7.01% Makefile 0.05% Jupyter Notebook 92.93%

genai llm benchmark streetfighterai

llm-colosseum's Introduction

Evaluate LLMs in real time with Street Fighter III

Make LLM fight each other in real time in Street Fighter III.

Which LLM will be the best fighter ?

Our criterias 🔥

They need to be:

Fast: It is a real time game, fast decisions are key
Smart: A good fighter thinks 50 moves ahead
Out of the box thinking: Outsmart your opponent with unexpected moves
Adaptable: Learn from your mistakes and adapt your strategy
Resilient: Keep your RPS high for an entire game

Let the fight begin 🥷

1 VS 1: Mistral 7B vs Mistral 7B

slow-v1.mp4

1 VS 1 X 6 : Mistral 7B vs Mistral 7B

6v6-fast.mp4

A new kind of benchmark ?

Street Fighter III assesses the ability of LLMs to understand their environment and take actions based on a specific context. As opposed to RL models, which blindly take actions based on the reward function, LLMs are fully aware of the context and act accordingly.

Results

Our experimentations (342 fights so far) led to the following leader board. Each LLM has an ELO score based on its results

Ranking

ELO ranking

Model	Rating
🥇openai:gpt-3.5-turbo-0125	1776.11
🥈mistral:mistral-small-latest	1586.16
🥉openai:gpt-4-1106-preview	1584.78
openai:gpt-4	1517.2
openai:gpt-4-turbo-preview	1509.28
openai:gpt-4-0125-preview	1438.92
mistral:mistral-medium-latest	1356.19
mistral:mistral-large-latest	1231.36

Win rate matrix

Explanation

Each player is controlled by an LLM. We send to the LLM a text description of the screen. The LLM decide on the next moves its character will make. The next moves depends on its previous moves, the moves of its opponents, its power and health bars.

Agent based
Multithreading
Real time

Installation

Follow instructions in https://docs.diambra.ai/#installation
Download the ROM and put it in ~/.diambra/roms
(Optional) Create and activate a new python venv
Install dependencies with make install or pip install -r requirements.txt
Create a .env file and fill it with the content like in the .env.example file
Run with make run

Test mode

To disable the LLM calls, set DISABLE_LLM to True in the .env file. It will choose the actions randomly.

Logging

Change the logging level in the script.py file.

Local model

You can run the arena with local models using Ollama.

Make sure you have ollama installed, running, and with a model downloaded (run ollama serve mistral in the terminal for example)
Run make local to start the fight.

By default, it runs mistral against mistral. To use other models, you need to change the parameter model in ollama.py.

from eval.game import Game, Player1, Player2

def main():
    game = Game(
        render=True,
        save_game=True,
        player_1=Player1(
            nickname="Baby",
            model="ollama:mistral", # change this
        ),
        player_2=Player2(
            nickname="Daddy",
            model="ollama:mistral", # change this
        ),
    )
    game.run()
    return 0

The convention we use is model_provider:model_name. If you want to use another local model than Mistral, you can do ollama:some_other_model

How to make my own LLM model play? Can I improve the prompts?

The LLM is called in Robot.call_llm() method of the agent/robot.py file.

    def call_llm(
        self,
        temperature: float = 0.7,
        max_tokens: int = 50,
        top_p: float = 1.0,
    ) -> str:
        """
        Make an API call to the language model.

        Edit this method to change the behavior of the robot!
        """
        # self.model is a slug like mistral:mistral-small-latest or ollama:mistral
        provider_name, model_name = get_provider_and_model(self.model)
        client = get_sync_client(provider_name) # OpenAI client

        # Generate the prompts
        move_list = "- " + "\n - ".join([move for move in META_INSTRUCTIONS])
        system_prompt = f"""You are the best and most aggressive Street Fighter III 3rd strike player in the world.
Your character is {self.character}. Your goal is to beat the other opponent. You respond with a bullet point list of moves.
{self.context_prompt()}
The moves you can use are:
{move_list}
----
Reply with a bullet point list of moves. The format should be: `- <name of the move>` separated by a new line.
Example if the opponent is close:
- Move closer
- Medium Punch

Example if the opponent is far:
- Fireball
- Move closer"""

        # Call the LLM
        completion = client.chat.completions.create(
            model=model_name,
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": "Your next moves are:"},
            ],
            temperature=temperature,
            max_tokens=max_tokens,
            top_p=top_p,
        )

        # Return the string to be parsed with regex
        llm_response = completion.choices[0].message.content.strip()
        return llm_response

To use another model or other prompts, make a call to another client in this function, change the system prompt, or make any fancy stuff.

Submit your model

Create a new class herited from Robot that has the changes you want to make and open a PR.

We'll do our best to add it to the ranking!

Credits

Made with ❤️ by the OpenGenerativeAI team from phospho (@oulianov @Pierre-LouisBJT @Platinn) and Quivr (@StanGirard) during Mistral Hackathon 2024 in San Francisco

llm-colosseum's People

Contributors

Stargazers

Watchers

Forkers

apollohuang1 truestone soxunlocks teganmosi rooben-me touristshaun tokkiu wayan123 techthiyanes cdrhim argo12 hscspring yacineali74 moritz-august gustavomacedo20 kswvl jiezh fkhawar shivak11 ishan-marikar manuelapetsi thanhpham1987 ystoll sadsasuke bananemure santosrai aws-banjo didlawowo angelicogfa ramstorageai dyanos adambear hhyqhh qima joon-stack josephrp zxser xbeark yimi81 robguilarr-dev contropist gaohuan2015 kankan1322 mivanovitch chandrath scan-dev kofawp dzineer miamicrypto waichan8 parisneo flyineye martyyz2112 octag0no utkarshx mbijon fangyuan-ksgk hannibalhuang cephren nikhillllllllll smith2t princetrunks veryvanya jesusoctavioas embeddedsamurai enoch3712 conqlab harry-cohere yeefeizzz miguelangelmy 54457616 taocao hasakiwan openchatgpts mysail shimura0 dkzdev loongz adityasoni17 sampink jatrott qqqqqwwwweeerr1gmail lizhongyi123 rickyhong allthingsllm iyanghsiao davidnguyenthang lalvarezcan doudoukiss multipolarityai hkabbay zhouping988 antonsaurytski83 vasiliyeskin southglory wandy666 arelkeselbri lhua0420 m0th3rch1p dsdanielpark

llm-colosseum's Issues

ELO ranking score?

How to generate this ranking? If I added new model, how to reproduce this benchmark?

ERROR after upgrade new docker image

@oulianov

`(streetfighter) taozhiyu@603e5f4a42f1 llm-colosseum % make local

diambra -r ~/.diambra/roms run -l python3 ollama.py

🖥️ Starting DIAMBRA environment:

🖥️ logged in

v2.2: Pulling from diambra/engine

cf584053a88a: Pull complete

Digest: sha256:77ba99e5d7d099fe2db292fc7b4e43da310e1bbdb0bcc3a157810e1f933ec81d

Status: Downloaded newer image for diambra/engine:v2.2

Stored credentials found.

Authorization granted.

Server listening on 0.0.0.0:50051

🖥️ DIAMBRA environment started

Traceback (most recent call last):

File "/Users/taozhiyu/Downloads/llm-colosseum/ollama.py", line 4, in

from eval.game import Game, Player1, Player2

File "/Users/taozhiyu/Downloads/llm-colosseum/eval/game.py", line 8, in

from agent import KEN_GREEN, KEN_RED, Robot

File "/Users/taozhiyu/Downloads/llm-colosseum/agent/init.py", line 6, in

from .robot import Robot

File "/Users/taozhiyu/Downloads/llm-colosseum/agent/robot.py", line 11, in

from llama_index.core.llms import ChatMessage

ModuleNotFoundError: No module named 'llama_index'

🖥️ Couldn't run: exit status 1

make: *** [local] Error 1

can not run local model

I installed ollama and follow the README.md. But can not run local model such as default option ollama:mistral.
error log:
Exception in thread Thread-5: Traceback (most recent call last): File "C:\Users\86138\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1009, in _bootstrap_inner self.run() File "D:\AI\llm-colosseum-git\eval\game.py", line 369, in run self.run() File "D:\AI\llm-colosseum-git\eval\game.py", line 383, in run self.game.player_1.robot.plan() File "D:\AI\llm-colosseum-git\agent\robot.py", line 134, in plan self.game.player_2.robot.plan() next_steps_from_llm = self.get_moves_from_llm() File "D:\AI\llm-colosseum-git\agent\robot.py", line 134, in plan File "D:\AI\llm-colosseum-git\agent\robot.py", line 293, in get_moves_from_llm next_steps_from_llm = self.get_moves_from_llm() File "D:\AI\llm-colosseum-git\agent\robot.py", line 293, in get_moves_from_llm llm_stream = self.call_llm() File "D:\AI\llm-colosseum-git\agent\robot.py", line 369, in call_llm llm_stream = self.call_llm() File "D:\AI\llm-colosseum-git\agent\robot.py", line 369, in call_llm resp = client.stream_chat(messages) AttributeError: 'NoneType' object has no attribute 'stream_chat' resp = client.stream_chat(messages) AttributeError: 'NoneType' object has no attribute 'stream_chat'

It seems robot.py:363 client = get_client(self.model) didn't get a valid model.
I checked the llm.py and in function get_client, there seems no branch to deal with "ollama:mistral" (provider is ollama)?
I wondered how the local model be created? Or I made some mistake?

if provider == "openai":
    from llama_index.llms.openai import OpenAI

    return OpenAI(model=model_name)
elif provider == "anthropic":
    from llama_index.llms.anthropic import Anthropic

    return Anthropic(model=model_name)
elif provider == "mixtral" or provider == "groq":
    from llama_index.llms.groq import Groq

    return Groq(model=model_name)

Hello, brother. How to modify the program so that AI can play computer controlled characters?

Thank you so much for this project, it was great!

But I don't want to have two AI's at the same time, I want to have AI's and computer characters.

larger model, worse peformance?

hi, the leader board shows arger model, worse peformance, is it because of the inference time? smaller model have high action frequency. if so, the bench is not very useful.

i think maybe change the game so it can pause, then we can compare models without bias on inference latency.

How to use Google gemini model

How to use Google gemini model to fighting?

MacOS only?

Downloading pyobjc-core-10.2.tar.gz (929 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 929.8/929.8 kB 14.6 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [2 lines of output]
running egg_info
error: PyObjC requires macOS to build
[end of output]

QWEN 1.8B or less is OK; 4B or more not working

no action generated, I think maybe token/per second is too low to run the game.

🏟️ (6e3b) (0)----------------------------------------
(0) Thank you for using DIAMBRA Arena.
(0) RL is the main road torwards AGI,
(0) better to be prepared... Keep rocking!
(0) -
(0) https://diambra.ai
(0)----------------------------------------
Player 1 using: ollama:qwen:1.8b-chat-v1.5-fp16
Player 2 using: ollama:qwen:4b-chat-v1.5-fp16
INFO:diambra.arena.engine.interface:Trying to connect to DIAMBRA Engine server (timeout=600s)...
INFO:diambra.arena.engine.interface:... done.
🏟️ (6e3b) (0)Overwriting sys_settings provided by the user with command line / CLI ones
(0)Provided by the user:

(0)Set via command line / CLI:
emu_pipes_path: "/tmp/DIAMBRA/"
roms_path: "/opt/diambraArena/roms/"
binary_path: "/opt/diambraArena/mame/"
lock_fps: true
username: "10545863"

  .:-:-**#*#####+***+=-:.

..-++####+#################=+=:.
:+############################-.
.-+#############################=.
.-###########################+.
.=######++======++##########=. ........ ... ....... ........... ........ ........ ............ ............ ...........
-=------:---------=########- .------..-----:. .------. .-----------. --------. :-------- --------------:. --------------:. :----------:
.:--------:---------:=#######* .------..-------..------. :-----:-----:. ---------. .--------- ------:..:-----: ------:..------: .------:-----.
.:------::---::--------:+#######..------. .------..------. .------.:-----. ----------.---------- ------:..------. ------: :-----. .:-----:.------.
:-----::.:---:-----::---:######* .------. .------..------. .:-----: .------. ------:-------------- ------::-----:.. ------:.-----:. .------..------.
.--:.. . :::.:.---.-:---.#####- .------. .------..------. .------.::------: ------:.-----.:------ ------: :-----: ------:.:------: .------:.:-------.
..:..:::.:::.::....:.::--:####. .------::------:..------..------:.:::------. ------: .---. :-----: -------::------: ------: ------: :------..::------:
.-::::::---...:--. .-.:-::=###+. .::::::::::::.. .::::::..::::::. .::::::. ::::::. .:. ::::::: ::::::::::::::. ::::::: ::::::: ::::::. .::::::
.--:. .:--...::. ..::+####:
:--:.. .---:...:. .:++:.
.:---:::----.. .... .:.
.:------:..:... :.
....... ..:. ..

                                                               DIAMBRA™ | Dueling AI Arena
                                                          https://diambra.ai - [email protected]

                               Usage of this software is subject to our Terms of Use described at https://diambra.ai/terms

                                                           DIAMBRA, Inc. © Copyright 2018-2024

(0)Environment initialization ...
🏟️ (6e3b) SHA256 check ok. Correct rom file found.
🏟️ (6e3b) Completed console init
🏟️ (6e3b) Fontconfig error: Cannot load default config file
🏟️ (6e3b) Warning: -video none doesn't make much sense without -seconds_to_run
🏟️ (6e3b) ALSA lib conf.c:4553:(snd_config_update_r) Cannot access file /usr/share/alsa/alsa.conf
ALSA lib seq.c:935:(snd_seq_open_noupdate) Unknown SEQ default
🏟️ (6e3b) Unable to create history.db
Unable to create history.db
Unable to create history.db
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
🏟️ (6e3b) Registering screen ... done.
🏟️ (6e3b) Registering audio ... done.
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: low punch
Player 1 move: super attack 2
Player 1 move: super attack 3
Player 1 move: super attack 4
Player 1 move: medium punch
Player 1 move: low punch
Player 1 move: low punch
2024-03-30 21:01:37.417 | WARNING | agent.robot:get_moves_from_llm:317 - Many invalid moves: ['High Attack', 'Low Attack', 'High Attack']
🏟️ (6e3b) Registering program ... done.
🏟️ (6e3b) Num. of Channels = 4
Screen Dim (W x H) = 384 224
🏟️ (6e3b) (Recorder) Frame encoding enabled.
(Recorder) Compression quality: 95
(0)Buttons configuration:
(0) LP = But4
(0) HP = But5
(0) MP = But1
(0) LK = But3
(0) MK = But2
(0) HK = But6
(0)Native frame shape = [224 X 384 X 4]
(0)User defined frame_shape = [0 X 0 X 0]
Resize flag = 0
Grayscale flag = 0
🏟️ (6e3b) (0)Move to start screen
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
🏟️ (6e3b) (0)Adjust game settings
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
🏟️ (6e3b) (0)done.
INFO:diambra.arena.arena_gym:EnvironmentSettingsMultiAgent(game_id='sfiii3n', frame_shape=[0, 0, 0], step_ratio=6, disable_keyboard=True, disable_joystick=True, render_mode='human', splash_screen=False, rank=0, env_address='127.0.0.1:55125', grpc_timeout=600, seed=1711803694, difficulty=None, continue_game=0.0, show_final=False, tower=3, _last_seed=1711803694, pb_model=game_id: "sfiii3n"
frame_shape {
}
step_ratio: 6
n_players: 2
disable_keyboard: true
disable_joystick: true
action_spaces: DISCRETE
action_spaces: DISCRETE
episode_settings {
}
, n_players=2, action_space=(1, 1), role=(None, None), characters=['Ken', 'Ken'], outfits=[1, 3], super_art=[3, 3], fighting_style=(None, None), ultimate_style=(None, None))
INFO:diambra.arena.arena_gym:Recording trajectories in "/Users/taozhiyu/Downloads/llm-colosseum/eval/diambra/episode_recording/sfiii3n/-/20240330210134"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
🏟️ (6e3b) (0)WARNING: only one outfit selected by agent_0 while using 2P mode. Note that if a character faces himself, it will use the next available outfit.
(0)2P Environment
(0)Generic episode settings ---
(0)Random seed: 42
(0)agent_0 Role: P1
(0)agent_0 Character(s): [Ken]
(0)agent_0 Number of outfits: 1
(0)agent_1 Role: P2
(0)agent_1 Character(s): [Ken]
(0)agent_1 Number of outfits: 1
(0)---
(0)Game-specific episode settings ---
(0)agent_0 Super art: 3
(0)agent_1 Super art: 3
(0)---
(0)Restarting system and (optionally) setting difficulty
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
🏟️ (6e3b) (0)Starting game
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
🏟️ (6e3b) (0)Waiting for fight to start
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: low punch
Player 1 move: high kick
Player 1 move: medium punch
Player 1 move: super attack 2
Player 1 move: super attack 3
Player 1 move: super attack 4
Player 1 move: jump closer
2024-03-30 21:02:42.124 | WARNING | agent.robot:get_moves_from_llm:317 - Many invalid moves: ['Super Attack 1', 'Super attack 5', 'Super attack']
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 2 move: jump closer
Player 2 move: low punch
Player 2 move: high punch
Player 2 move: medium punch
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: fireball
Player 1 move: low kick
Player 1 move: high punch
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: hurricane
Player 1 move: megafireball
Player 1 move: super attack 2
Player 1 move: super attack 3
Player 1 move: super attack 4
Player 1 move: low punch
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: low kick
Player 1 move: high punch
Player 1 move: medium punch
Player 1 move: super attack 2
Player 1 move: super attack 3
Player 1 move: super attack 4
Player 1 move: low punch
Player 1 move: low punch
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: super attack 2
Player 1 move: high punch
Player 1 move: low punch
Player 1 move: medium punch
Player 1 move: high punch
Player 1 move: jump closer
Player 1 move: jump away
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: hurricane
Player 1 move: super attack 2
Player 1 move: jump closer
Player 1 move: low punch
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: low punch
Player 1 move: super attack 2
Player 1 move: high kick
Player 1 move: megafireball
Player 1 move: super attack 3
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: super attack 3
Player 1 move: super attack 4
Player 1 move: low punch
Player 1 move: hurricane
Player 1 move: megafireball
Player 1 move: medium punch
Player 1 move: jump closer
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: super attack 4
2024-03-30 21:04:08.569 | WARNING | agent.robot:get_moves_from_llm:317 - Many invalid moves: ['Low Strike 1 ', 'Low Strike 2 ']
🏟️ (6e3b) (0)Round won by P1
(0)Moving to next round
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player1 ollama:qwen:1.8b-chat-v1.5-fp16 'Baby' won!
🏟️ (6e3b) Closing console
🏟️ (6e3b) (0)----------------------------------------
(0) Thank you for using DIAMBRA Arena.
(0) RL is the main road torwards AGI,
(0) better to be prepared... Keep rocking!
(0) -
(0) https://diambra.ai
(0)----------------------------------------
Player 1 using: ollama:qwen:1.8b-chat-v1.5-fp16
Player 2 using: ollama:qwen:4b-chat-v1.5-fp16
INFO:diambra.arena.engine.interface:Trying to connect to DIAMBRA Engine server (timeout=600s)...
INFO:diambra.arena.engine.interface:... done.
🏟️ (6e3b) (0)Overwriting sys_settings provided by the user with command line / CLI ones
(0)Provided by the user:

(0)Set via command line / CLI:
emu_pipes_path: "/tmp/DIAMBRA/"
roms_path: "/opt/diambraArena/roms/"
binary_path: "/opt/diambraArena/mame/"
lock_fps: true
username: "10545863"

  .:-:-**#*#####+***+=-:.

                                                               DIAMBRA™ | Dueling AI Arena
                                                          https://diambra.ai - [email protected]

                               Usage of this software is subject to our Terms of Use described at https://diambra.ai/terms

                                                           DIAMBRA, Inc. © Copyright 2018-2024

(streetfighter) taozhiyu@192 llm-colosseum % ollama list
NAME ID SIZE MODIFIED
qwen:0.5b-chat-v1.5-fp16 967f7a3593ba 1.2 GB 11 hours ago
qwen:1.8b-chat-v1.5-fp16 e3562f7740ef 3.7 GB 7 hours ago
qwen:14b-chat-v1.5-fp16 cb20f077361d 28 GB 4 days ago
qwen:4b-chat-v1.5-fp16 86621ca225c4 7.9 GB 7 hours ago
qwen:7b-chat-v1.5-fp16 39e2b1482d7d 15 GB 7 hours ago
(streetfighter) taozhiyu@192 llm-colosseum % make local
diambra -r ~/.diambra/roms run -l python3 ollama.py
🖥️ Starting DIAMBRA environment:
🖥️ logged in
v2.2: Pulling from diambra/engine
Digest: sha256:6b5df5c9522553a4505bf0b6b0f837dd2f945de1b0b6390d83fdf24e317de643
Status: Image is up to date for diambra/engine:v2.2
Stored credentials found.
Authorization granted.
Server listening on 0.0.0.0:50051
🖥️ DIAMBRA environment started
Player 1 using: ollama:qwen:4b-chat-v1.5-fp16
Player 2 using: ollama:qwen:4b-chat-v1.5-fp16
INFO:diambra.arena.engine.interface:Trying to connect to DIAMBRA Engine server (timeout=600s)...
INFO:diambra.arena.engine.interface:... done.
🏟️ (6c62) (0)Overwriting sys_settings provided by the user with command line / CLI ones
(0)Provided by the user:
🏟️ (6c62)
(0)Set via command line / CLI:
emu_pipes_path: "/tmp/DIAMBRA/"
roms_path: "/opt/diambraArena/roms/"
binary_path: "/opt/diambraArena/mame/"
lock_fps: true
username: "10545863"

  .:-:-**#*#####+***+=-:.

                                                               DIAMBRA™ | Dueling AI Arena
                                                          https://diambra.ai - [email protected]

                               Usage of this software is subject to our Terms of Use described at https://diambra.ai/terms

                                                           DIAMBRA, Inc. © Copyright 2018-2024

(0)Environment initialization ...
🏟️ (6c62) SHA256 check ok. Correct rom file found.
🏟️ (6c62) Completed console init
🏟️ (6c62) Fontconfig error: Cannot load default config file
🏟️ (6c62) Warning: -video none doesn't make much sense without -seconds_to_run
🏟️ (6c62) ALSA lib conf.c:4553:(snd_config_update_r) Cannot access file /usr/share/alsa/alsa.conf
ALSA lib seq.c:935:(snd_seq_open_noupdate) Unknown SEQ default
🏟️ (6c62) Unable to create history.db
Unable to create history.db
🏟️ (6c62) Unable to create history.db
🏟️ (6c62) Registering screen ... done.
🏟️ (6c62) Registering audio ... done.
🏟️ (6c62) Registering program ... done.
🏟️ (6c62) Num. of Channels = 4
Screen Dim (W x H) = 384 224
🏟️ (6c62) (Recorder) Frame encoding enabled.
(Recorder) Compression quality: 95
(0)Buttons configuration:
(0) LP = But4
(0) HK = But6
(0) MK = But2
(0) HP = But5
(0) MP = But1
(0) LK = But3
🏟️ (6c62) (0)Native frame shape = [224 X 384 X 4]
(0)User defined frame_shape = [0 X 0 X 0]
Resize flag = 0
Grayscale flag = 0
🏟️ (6c62) (0)Move to start screen
🏟️ (6c62) (0)Adjust game settings
🏟️ (6c62) (0)done.
INFO:diambra.arena.arena_gym:EnvironmentSettingsMultiAgent(game_id='sfiii3n', frame_shape=[0, 0, 0], step_ratio=6, disable_keyboard=True, disable_joystick=True, render_mode='human', splash_screen=False, rank=0, env_address='127.0.0.1:55126', grpc_timeout=600, seed=1711803921, difficulty=None, continue_game=0.0, show_final=False, tower=3, _last_seed=1711803921, pb_model=game_id: "sfiii3n"
frame_shape {
}
step_ratio: 6
n_players: 2
disable_keyboard: true
disable_joystick: true
action_spaces: DISCRETE
action_spaces: DISCRETE
episode_settings {
}
, n_players=2, action_space=(1, 1), role=(None, None), characters=['Ken', 'Ken'], outfits=[1, 3], super_art=[3, 3], fighting_style=(None, None), ultimate_style=(None, None))
INFO:diambra.arena.arena_gym:Recording trajectories in "/Users/taozhiyu/Downloads/llm-colosseum/eval/diambra/episode_recording/sfiii3n/-/20240330210521"
🏟️ (6c62) (0)WARNING: only one outfit selected by agent_0 while using 2P mode. Note that if a character faces himself, it will use the next available outfit.
(0)2P Environment
(0)Generic episode settings ---
(0)Random seed: 42
(0)agent_0 Role: P1
(0)agent_0 Character(s): [Ken]
(0)agent_0 Number of outfits: 1
(0)agent_1 Role: P2
(0)agent_1 Character(s): [Ken]
(0)agent_1 Number of outfits: 1
(0)---
(0)Game-specific episode settings ---
(0)agent_0 Super art: 3
(0)agent_1 Super art: 3
(0)---
(0)Restarting system and (optionally) setting difficulty
🏟️ (6c62) (0)Starting game
🏟️ (6c62) (0)Waiting for fight to start
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 2 move: jump away
Player 2 move: high punch
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
Player 1 move: move away
Player 1 move: low punch
Player 1 move: medium punch
Player 1 move: high punch
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
🖥️ Received signal, terminating

Unset API keys are not handled properly

I noticed that in the current main branch, mistral is always activated by default. That of course causes a problem if the API key is not actually set. Similar case also for the OpenAI key.

In you case you don't already have a fix for that in some branch, I could create a PR for that.

how to change characters?

it seems Qwen 0.5B is better model to run. by the way, how to change characters?

Alex, Twelve, Hugo, Sean, Makoto, Elena, Ibuki, Chun-Li, Dudley, Necro, Q, Oro, Urien, Remy, Ryu, Gouki, Yun, Yang, Ken

Is there a way to set it to do best 3 of 5?

Is there a way to get it to set the matches to a best of...
3 of 5
4 of 7, etc

Thanks,

Issue while trying to run locally

Hello!
First, thank you for the interesting project!
I am trying to run it locally with ollama using an anaconda environment. However, I am getting the following error:

File "\llmtest\llm-colosseum\agent\robot.py", line 135, in plan
KeyError: 12
next_buttons_to_press = [
File "\llmtest\llm-colosseum\agent\robot.py", line 138, in
for button in META_INSTRUCTIONS_WITH_LOWER[combo][self.current_direction.lower()]
KeyError: 6

Context: The game window appears, but the characters do not do anything.

Any idea on what is happening here?

NEW DOCKER IMAGE ERROR: fighter starts, but two Kens do not fight.

fighter starts, but two Kens do not fight.
@oulianov
'
(0)Restarting system and (optionally) setting difficulty
🏟️ (864b) (0)Starting game
🏟️ (864b) (0)Waiting for fight to start
Exception in thread Thread-5:
Exception in thread Thread-6:
Traceback (most recent call last):
File "/Users/taozhiyu/miniconda3/envs/streetfighter/lib/python3.9/threading.py", line 980, in _bootstrap_inner
Traceback (most recent call last):
File "/Users/taozhiyu/miniconda3/envs/streetfighter/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/Users/taozhiyu/Downloads/llm-colosseum/eval/game.py", line 369, in run
self.run()
File "/Users/taozhiyu/Downloads/llm-colosseum/eval/game.py", line 383, in run
self.game.player_2.robot.plan()
File "/Users/taozhiyu/Downloads/llm-colosseum/agent/robot.py", line 134, in plan
self.game.player_1.robot.plan()
File "/Users/taozhiyu/Downloads/llm-colosseum/agent/robot.py", line 134, in plan
next_steps_from_llm = self.get_moves_from_llm()
File "/Users/taozhiyu/Downloads/llm-colosseum/agent/robot.py", line 293, in get_moves_from_llm
next_steps_from_llm = self.get_moves_from_llm()
File "/Users/taozhiyu/Downloads/llm-colosseum/agent/robot.py", line 293, in get_moves_from_llm
llm_stream = self.call_llm()
File "/Users/taozhiyu/Downloads/llm-colosseum/agent/robot.py", line 369, in call_llm
llm_stream = self.call_llm()
File "/Users/taozhiyu/Downloads/llm-colosseum/agent/robot.py", line 369, in call_llm
resp = client.stream_chat(messages)
AttributeError: 'NoneType' object has no attribute 'stream_chat'
resp = client.stream_chat(messages)
AttributeError: 'NoneType' object has no attribute 'stream_chat'
'

Add Google Gemini model

fight in loop

@oulianov

if name == "main":
for _ in range(10):
main()

modify code like this. automatically fight in loop. ^_^

Report Different models fight on the street

two models fight 50 rounds, the report is below

KeyError

Hi, I have the following issue:
llm-colosseum/agent/robot.py", line 137, in
for button in META_INSTRUCTIONS_WITH_LOWER[combo][
for button in META_INSTRUCTIONS_WITH_LOWER[combo][
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 6
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 12

under macos m1 and local mistral, here an extract of my setup.py:
player_1=Player1(
nickname="Daddy",
#model=generate_random_model(mistral=True),
model="ollama:mistral",
),
player_2=Player2(
nickname="Baby",
#model=generate_random_model(openai=True),
model="ollama:mistral",
),

The game is displayed countdown running but both fighters stays idle.
Thanks

how to set show_final=true???

INFO:diambra.arena.arena_gym:EnvironmentSettingsMultiAgent(game_id='sfiii3n', frame_shape=[0, 0, 0], step_ratio=6, disable_keyboard=True, disable_joystick=True, render_mode='human', splash_screen=False, rank=0, env_address='127.0.0.1:55006', grpc_timeout=600, seed=1712025299, difficulty=None, continue_game=0.0, show_final=False, tower=3, _last_seed=1712025299, pb_model=game_id: "sfiii3n"

how to set show_final=true???

I try to look for in it py files. but not found.

can't fetch new commit

look. GitHub desktop version can not get your latest commit

![WX20240510-203802@2x](https://github.com/OpenGenerativeAI/llm-colosseum/assets/146583103/9ab3dcf4-e82e-46f7-9a3b-54a5284c9a96)

Yi 6b , no action

it is yi:6b-fp16 model for two players. why does it not generate any actions?

[question]Two characters cannot approach each other after they switch positions.

I replace the second character with Gouki and define it as white Gouki_WHITE = [255, 255, 255]. If the two characters do not switch positions, the game proceeds normally, but once the two characters switch positions, the two characters cannot approach each other. play1 KEN_RED = [248, 0, 0], play 2 Gouki_WHITE = [255, 255, 255].
Please point out my programming error.

robot.py:

import os
import random
import re
import time
from collections import defaultdict
from typing import Dict, List, Literal, Optional

import numpy as np
from gymnasium import spaces
from loguru import logger
from phospho.lab import get_provider_and_model, get_sync_client
from rich import print

from .config import (
INDEX_TO_MOVE,
META_INSTRUCTIONS,
META_INSTRUCTIONS_WITH_LOWER,
MOVES,
NB_FRAME_WAIT,
X_SIZE,
Y_SIZE,
)
from .observer import detect_position_from_color

class Robot:
observations: List[Optional[dict]] = None # memory
next_steps: List[int] # action plan
actions: dict # actions of the agents during a step of the game
# actions of the agents during the previous step of the game
previous_actions: Dict[str, List[int]]
reward: float # reward of the agent

action_space: spaces.Space
character: Optional[str] = None  # character name
side: int  # side of the stage where playing: 0 = left, 1 = right
current_direction: Literal["Left", "Right"]  # current direction facing
sleepy: Optional[bool] = False  # if the robot is sleepy
only_punch: Optional[bool] = False  # if the robot only punch

model: str  # model of the robot
super_bar_own: int
player_nb: int  # player number

def __init__(
    self,
    action_space: spaces.Space,
    character: str,
    side: int,
    character_color: list,
    ennemy_color: list,
    sleepy: bool = False,
    only_punch: bool = False,
    model: str = "mistral:mistral-large-latest",
    player_nb: int = 0,  # 0 means not specified
):
    self.action_space = action_space
    self.character = character
    if side == 0:
        self.current_direction = "Right"
    elif side == 1:
        self.current_direction = "Left"

    self.observations = []
    self.next_steps = []
    self.character_color = character_color  # Character color list
    self.ennemy_color = ennemy_color  # ennemy color list
    self.side = side
    self.sleepy = sleepy
    self.only_punch = only_punch
    self.model = model
    self.previous_actions = defaultdict(list)
    self.actions = {}
    self.player_nb = player_nb

def act(self) -> int:
    """
    At each game frame, we execute the first action in the list of next steps.

    An action is an integer from 0 to 18, where 0 is no action.

    See the MOVES dictionary for the mapping of actions to moves.
    """
    if not self.next_steps or len(self.next_steps) == 0:
        return 0  # No move

    if self.sleepy:
        return 0

    if self.only_punch:
        # Do a Hadouken
        if self.current_direction == "Right":
            self.next_steps.extend(
                [
                    MOVES["Down"],
                    MOVES["Right+Down"],
                    MOVES["Right"],
                    MOVES["High Punch"],
                ]
            )
        elif self.current_direction == "Left":
            self.next_steps.extend(
                [
                    MOVES["Down"],
                    MOVES["Down+Left"],
                    MOVES["Left"],
                    MOVES["High Punch"],
                ]
            )

    next_step = self.next_steps.pop(0)

    return next_step

def plan(self) -> None:
    """
    The robot will plan its next steps by calling this method.

    In SF3, moves are based on combos, which are list of actions that must be executed in a sequence.

    Moves of Ken
    https://www.eventhubs.com/guides/2009/may/11/ken-street-fighter-3-third-strike-character-guide/

    Moves of Ryu
    https://www.eventhubs.com/guides/2008/may/09/ryu-street-fighter-3-third-strike-character-guide/
    """

    # If we already have a next step, we don't need to plan
    if len(self.next_steps) > 0:
        return

    # Call the LLM to get the next steps
    next_steps_from_llm = self.get_moves_from_llm()
    next_buttons_to_press = [
        button
        for combo in next_steps_from_llm
        for button in META_INSTRUCTIONS_WITH_LOWER[combo][
            self.current_direction.lower()
        ]
        # We add a wait time after each button press
        + [0] * NB_FRAME_WAIT
    ]
    self.next_steps.extend(next_buttons_to_press)

def observe(self, observation: dict, actions: dict, reward: float):
    """
    The robot will observe the environment by calling this method.

    The latest observations are at the end of the list.
    """

    # detect the position of characters and ennemy based on color
    character_position = detect_position_from_color(
        observation, self.character_color
    )
    ennemy_position = detect_position_from_color(
        observation, self.ennemy_color
    )

    if character_position is not None and ennemy_position is not None:
        # Store the positions in the observation dictionary
        observation["character_position"] = character_position
        observation["ennemy_position"] = ennemy_position
    else:
        # If positions couldn't be detected, set them to None
        observation["character_position"] = None
        observation["ennemy_position"] = None

    self.observations.append(observation)
    # we delete the oldest observation if we have more than 10 observations
    if len(self.observations) > 10:
        self.observations.pop(0)

    self.reward = reward

    # Update previous actions
    for key, value in actions.items():
        if value != 0:
            self.previous_actions[key].append(value)
            if len(self.previous_actions[key]) > 10:
                self.previous_actions[key].pop(0)

    # Update current direction based on character and ennemy positions
    self.update_current_direction(observation)

def update_current_direction(self, observation):
    """
    Update the current direction based on character and ennemy positions.
    """
    character_position = observation.get("character_position")
    ennemy_position = observation.get("ennemy_position")
    if character_position is not None and ennemy_position is not None:
        if character_position[0] < ennemy_position[0]:
            self.current_direction = "Right"
        else:
            self.current_direction = "Left"

def context_prompt(self) -> str:
    """
    Return a str of the context

    "The observation for you is Left"
    "The observation for the opponent is Left+Up"
    "The action history is Up"
    """

    # Create the position prompt
    side = self.side
    obs_own = self.observations[-1]["character_position"]
    obs_opp = self.observations[-1]["ennemy_position"]
    super_bar_own = self.observations[-1]["P" + str(side + 1)]["super_bar"][0]

    position_prompt = ""
    if obs_own is not None and obs_opp is not None:
        own_x, own_y = obs_own
        opp_x, opp_y = obs_opp

        distance_x = opp_x - own_x
        if distance_x < 0:
            position_prompt += "Your opponent is on the left."
        elif distance_x > 0:
            position_prompt += "Your opponent is on the right."
        else:
            position_prompt += "You are very close to the opponent."
    else:
        position_prompt = "Unable to determine opponent's position."

    power_prompt = ""
    if super_bar_own >= 30:
        power_prompt = "You can now use a powerful move. The names of the powerful moves are: Megafireball, Super attack 2."
    if super_bar_own >= 120 or super_bar_own == 0:
        power_prompt = "You can now only use very powerful moves. The names of the very powerful moves are: Super attack 3, Super attack 4"

    last_action_prompt = ""
    if len(self.previous_actions.keys()) > 0:
        act_own_list = self.previous_actions["agent_" + str(side)]
        act_opp_list = self.previous_actions["agent_" + str(abs(1 - side))]

        if len(act_own_list) == 0:
            act_own = 0
        else:
            act_own = act_own_list[-1]
        if len(act_opp_list) == 0:
            act_opp = 0
        else:
            act_opp = act_opp_list[-1]

        str_act_own = INDEX_TO_MOVE.get(act_own, "No action")
        str_act_opp = INDEX_TO_MOVE.get(act_opp, "No action")

        last_action_prompt += f"Your last action was {str_act_own}. The opponent's last action was {str_act_opp}."

    reward = self.reward

    score_prompt = ""
    if reward > 0:
        score_prompt += "You are winning. Keep attacking the opponent."
    elif reward < 0:
        score_prompt += "You are losing. Continue to attack the opponent but don't get hit."

    context = f"""{position_prompt}

{power_prompt}
{last_action_prompt}
Your current score is {reward}. {score_prompt}
To increase your score, move toward the opponent and attack the opponent. To prevent your score from decreasing, don't get hit by the opponent.
"""

    return context

def get_moves_from_llm(
    self,
) -> List[str]:
    """
    Get a list of moves from the language model.
    """

    # Filter the moves that are not in the list of moves
    invalid_moves = []
    valid_moves = []

    # If we are in the test environment, we don't want to call the LLM
    if os.getenv("DISABLE_LLM", "False") == "True":
        # Choose a random int from the list of moves
        logger.debug("DISABLE_LLM is True, returning a random move")
        return [random.choice(list(MOVES.values()))]

    while len(valid_moves) == 0:
        llm_response = self.call_llm()

        # The response is a bullet point list of moves. Use regex
        matches = re.findall(r"- ([\w ]+)", llm_response)
        moves = ["".join(match) for match in matches]
        invalid_moves = []
        valid_moves = []
        for move in moves:
            cleaned_move_name = move.strip().lower()
            if cleaned_move_name in META_INSTRUCTIONS_WITH_LOWER.keys():
                if self.player_nb == 1:
                    print(
                        f"[red] Player {self.player_nb} move: {cleaned_move_name}"
                    )
                elif self.player_nb == 2:
                    print(
                        f"[green] Player {self.player_nb} move: {cleaned_move_name}"
                    )
                valid_moves.append(cleaned_move_name)
            else:
                logger.debug(f"Invalid completion: {move}")
                logger.debug(f"Cleaned move name: {cleaned_move_name}")
                invalid_moves.append(move)

        if len(invalid_moves) > 1:
            logger.warning(f"Many invalid moves: {invalid_moves}")

    logger.debug(f"Next moves: {valid_moves}")
    return valid_moves

def call_llm(
    self,
    temperature: float = 0.7,
    max_tokens: int = 50,
    top_p: float = 1.0,
) -> str:
    """
    Make an API call to the language model.

    Edit this method to change the behavior of the robot!
    """
    provider_name, model_name = get_provider_and_model(self.model)
    client = get_sync_client(provider_name)

    # Generate the prompts
    move_list = "- " + "\n - ".join([move for move in META_INSTRUCTIONS])
    system_prompt = f"""You are the best and most aggressive Street Fighter III 3rd strike player in the world.

Your character is {self.character}. Your goal is to defeat the opponent. While constantly approaching and striking your opponent, you should also pay attention to your defense. If the opponent attacks you with moves like Fireball, you should retreat to defend, then seize the opportunity to close in and launch a deadly attack on your opponent.
{self.context_prompt()}
The moves you can use are:
{move_list}

Reply with a bullet point list of moves. The format should be: - <name of the move> separated by a new line.
Example if the opponent is close:

Move closer
Medium Punch

Example if the opponent is far:

Fireball

Move closer"""

  start_time = time.time()
  completion = client.chat.completions.create(
      model=model_name,
      messages=[
          {"role": "system", "content": system_prompt},
          {"role": "user", "content": "Your next moves are:"},
      ],
      temperature=temperature,
      max_tokens=max_tokens,
      top_p=top_p,
  )
  logger.debug(f"LLM call to {self.model}: {system_prompt}")
  logger.debug(f"LLM call to {self.model}: {time.time() - start_time}s")
  llm_response = completion.choices[0].message.content.strip()
  return llm_response

can I run locally?

I plan to "ollama serve" two different models locally, how should I set .env? and base URL?

`MISTRAL_API_KEY=""

OPENAI_API_KEY="ollama"

GROK_API_KEY=""

MODEL_PROVIDER="openai" # Options: ["openai", "mistral"]

DISABLE_LLM="False"`

🏟️ (eb4e) Warning: -video none doesn't make much sense without -seconds_to_run

🏟️ (eb4e) ALSA lib conf.c:4553:(snd_config_update_r) Cannot access file /usr/share/alsa/alsa.conf

ALSA lib seq.c:935:(snd_seq_open_noupdate) Unknown SEQ default

🏟️ (eb4e) Unable to create history.db

Unable to create history.db

🏟️ (eb4e) Unable to create history.db
`

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.