microsoft / jarvis Goto Github PK

View Code? Open in Web Editor NEW

23.1K 386.0 1.9K 36.77 MB

JARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf

License: MIT License

Python 90.20% Shell 0.80% JavaScript 0.69% HTML 0.11% Vue 5.33% TypeScript 2.13% CSS 0.17% PowerShell 0.58%

deep-learning platform pytorch

jarvis's People

Stargazers

Watchers

Forkers

tan-xu hertera1 change-maker touristshaun haikuoxin evelynmitchell emptydiagram atlas3dss jamal-0x1 bbouldin lowdias whitebaby restevesd logikonline quantumtau tony163163 therzv kustomzone codeaudit voltek62 monkofeast singlag lujohn74 apollohuang1 cloudbee7 mbyase lannerate zhangbillion npv2k1 maverick7 hannderson2002 eltociear ichbingautam thezeivier stjordanis goswamig srinathjukanti chan0415 edsun3941 walkerpnd xson2643 standardgalactic po-vincent jaedukseo aliang-chatgpt hhy5277 tearrain clouds233 hongwen-sun tonyabracadabra redpintings johnaffolter startime-h aikun7777777 yihangya bsli shanhedian2017 stepneverstop shendiaomo tuanbc dd-guo mengmengtech yizhangliu jackstephen keyman9848 cloudenginehub ovikk thirtyai zhenghfei rexiyz fengzifrank sienahwang strongerfly kingliantop herobrine19 code-arnab ailabteam wekingchen2077 techventurebuilder techthiyanes zhouqihong panzz cc6dll weexp brainhub24 kzke isaka dzbwhut ironnman yamseyoung rocksen d34d633f zzliekkas iamleon121 mmourafiq autogyro cgoder adrian-oprea vikrantsingh004 aryanutkarsh

jarvis's Issues

Getting no response when using lite.yaml as config for server mode

I'm not getting any responses when using lite yaml "python awesome_chat.py --config lite.yaml --mode server" , CLI mode works fine tho, wondering if anyone encontered the same issue?

Petition to backronym JARVIS as Just Analyzing Responses, Verifying Input, Synthesizing

I strongly believe that JARVIS deserves its name, and here's why:

Just Analyzing Responses: The core functionality of JARVIS lies in its ability to analyze responses generated by ChatGPT. The library efficiently handles user queries, dives deep into the knowledge base, and makes sense of the generated responses. This simplifies the overall process of interacting with ChatGPT and allows developers to focus on building creative applications.

Verifying Input: JARVIS ensures that the input provided to the language models is processed and verified. It checks for any potential errors or inconsistencies before sending the data to the models. This feature is crucial for maintaining the quality and accuracy of the output generated by ChatGPT.

Synthesizing: The most notable aspect of JARVIS is its ability to synthesize information. It can combine outputs from multiple language models and generate coherent, contextually relevant responses. This makes JARVIS an ideal tool for developers looking to create advanced natural language processing applications.

By harnessing the power of both ChatGPT and HuggingFace, JARVIS provides an unparalleled foundation for developers to build cutting-edge NLP solutions. The library is designed with a focus on simplicity, efficiency, and robustness, making it an ideal choice for those who wish to create versatile AI-driven applications.

Furthermore, JARVIS is open-source and designed to be accessible to developers of all skill levels. We encourage you to contribute to the project, share your ideas, and help improve the library, making it even more powerful and versatile.

error for help

python models_server.py --config config.yaml

Traceback (most recent call last):
File "models_server.py", line 23, in
import torchaudio
File "/Users/qinxiaoqiang/miniconda3/envs/jarvis/lib/python3.8/site-packages/torchaudio/init.py", line 1, in
from torchaudio import ( # noqa: F401
File "/Users/qinxiaoqiang/miniconda3/envs/jarvis/lib/python3.8/site-packages/torchaudio/_extension/init.py", line 5, in
from torchaudio._internal.module_utils import fail_with_message, is_module_available, no_op

ImportError: cannot import name 'fail_with_message' from 'torchaudio._internal.module_utils'

please generate an image where a high-speed train running at high speed.Then please describe the new image with your voice.

Consider using huggingface_hub library for Inference

In awesome_chat.py, you implement the inference method huggingface_model_inference (that uses the Hugging Face Hub under the hood) using requests and custom code. One thing you can do to deduplicate some code is to use huggingface_hub library and in particular its InferenceAPI class. You can find a guide here on how to use it.

For example, you can replace:

if task == "text-to-image":
    text = data["text"]
    response = requests.post(task_url, headers=HUGGINGFACE_HEADERS, json={"inputs": text})
    img_data = response.content
    img = Image.open(BytesIO(img_data))

if task == "text-to-image":
    img = InferenceApi(repo_id=model_id, task=task, token=token)(inputs=data["text"])

Some advantages of using huggingface_hub are:

it checks if model and task are compatible
if the user is logged in using huggingface-cli login, the token is automatically retrieved
(limited) some data parsing when possible (image => PIL)

Disclaimer: I am a maintainer of huggingface_hub. If you need any help with the integration, I'd be happy to help! 😃

建议发展一个替代chatgpt的选择，避免过早被拒访问而影响开源进展

据以往经验，为chatgpt这种在国内访问已被拒的生态组件做项目，如果做得好，都会受到审查牵连，影响后面的商业化进程，不是能翻就ok的，望重视早日落地成功

ModuleNotFoundError: No module named 'diffusers'

On windows 11
16 GB RAM
12th Gen Intel(R) Core(TM) i7-12700F
NVIDIA GeForce RTx3070

Command: python awesome_chat.py --config config.yaml --mode cli

Error:
File ".\JARVIS\server\awesome_chat.py", line 17, in
from diffusers.utils import load_image
ModuleNotFoundError: No module named 'diffusers'

april fools ?

weird feeling that is a joke, looking forward for the code

Error loading model whisper-base

While trying to run the models server I run into the following error:

raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
ValueError: Could not load model models/openai/whisper-base with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCTC'>, <class 'transformers.models.auto.modeling_auto.AutoModelForSpeechSeq2Seq'>, <class 'transformers.models.whisper.modeling_whisper.WhisperForConditionalGeneration'>).

I tried redoing all the installation steps, removing and recreating the conda env, and making sure all python dependencies are installed and both api tokens configured. What could be going wrong?

Error when start web server.

(jarvis) m200@DESKTOP-K0LFJU8:~/workdir/JARVIS/web$ npm run dev

> [email protected] dev
> vite

file:///home/m200/workdir/JARVIS/web/node_modules/vite/bin/vite.js:7
    await import('source-map-support').then((r) => r.default.install())
    ^^^^^

SyntaxError: Unexpected reserved word
    at Loader.moduleStrategy (internal/modules/esm/translators.js:133:18)
    at async link (internal/modules/esm/module_job.js:42:21)

Any idea?

Gradio. Built-in example - Joke and Cat. json.decoder.JSONDecodeError: Invalid \escape

Server is up an running. Using gradio python run_gradio_demo.py --config config.gradio.yaml:

Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

Running built-in example show me a joke and an image of cat. Gradio terminal output:

2023-04-06 15:45:43,407 - awesome_chat - INFO - ********************************************************************************
2023-04-06 15:45:43,407 - awesome_chat - INFO - input: show me a joke and an image of cat
2023-04-06 15:45:45,739 - awesome_chat - INFO - [{"task": "conversational", "id": 0, "dep": [-1], "args": {"text": "please show me a joke of cat" }}, {"task": "text-to-image", "id": 1, "dep": [-1], "args": {"text": "a photo of cat" }}]
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/routes.py", line 393, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 1108, in process_api
    result = await self.call_function(
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 915, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "run_gradio_demo.py", line 81, in bot
    message = chat_huggingface(all_messages, OPENAI_KEY)
  File "/home/user/Developer/JARVIS/server/awesome_chat.py", line 908, in chat_huggingface
    response = response_results(input, results, openaikey).strip()
  File "/home/user/Developer/JARVIS/server/awesome_chat.py", line 342, in response_results
    messages = json.loads(demos_or_presteps)
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid \escape: line 8 column 398 (char 527)

OSError: [Errno 99] Cannot assign requested address

When I run the models_server.py in aws , OSError: [Errno 99] Cannot assign requested address.
How can I deploy the service on the cloud server and I download all model in cloud .
And if i set config is hugging face run the server , "However, the inference result was an error, as the service related to ControlNet is not available."

Error:OSError: You seem to have cloned a repository without having git-lfs installed.

Encounter an error:

Error:OSError: You seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run `git lfs install` followed by `git lfs pull` in the folder you cloned.

How to solve this problem when running

python models_server.py --config config.yaml

Does this support multi-gpus?

have not a >48g gpu....

Is nvidia 4070ti enough to run this model?

I don't quite understand the requirement being rtx 3090.
Is it the vram that matters?

Can't run JARVIS in local full mode.

I downloaded all the models (28 models in 431GB) on my PC.
And I have hybrid/minimal mode running successfully.
But I can't have JARVIS running in local/full mode. (I do have 128GB RAM)
I got a error message saying that it can't load the file that exists actually. And I checked the file read permission with the process running user no problem.

Create dockerfile

Please include dockerfile or other isolated OS environment for improved building. If I get time this or next weekend, I'll try to create one and open a PR, otherwise please tag me if one gets created. Thanks!

项目可以跑了吗？

麻烦问一下各位，现在项目完善好了吗？有一块3090,16GB运存，不知道能带的动不？

KeyError: 'choices'

I am getting this error when using the cli:

Traceback (most recent call last):
File "awesome_chat.py", line 970, in
cli()
File "awesome_chat.py", line 941, in cli
answer = chat_huggingface(messages)
File "awesome_chat.py", line 850, in chat_huggingface
task_str = parse_task(context, input).strip()
File "awesome_chat.py", line 297, in parse_task
return send_request(data)
File "awesome_chat.py", line 167, in send_request
return response.json()["choices"][0]["text"].strip()
KeyError: 'choices'
(jarvis)

My prompt was this:
[ User ]: based on this image, assets\example_image.JPG , please tell me what you see.

Should we filter some useful models? Hard to generate any image or find one useful model.

config:

inference_mode: huggingface
local_deployment: minimal

If I run command:

python3 awesome_chat.py --config config.yaml --mode cli

I got these results:

[ User ]: show me a joke and a image about cat
[ Jarvis ]:  I understand your request. Firstly, I used the GPT-2 model to generate a joke for you. Unfortunately, there was an inference error and I could not generate a joke. Secondly, I used the Google/VIT-Base-Patch16-224 model to classify an image of a cat. However, there was an error in the path or URL, so I could not generate an image. Lastly, I used the PromptHero/OpenJourney model to generate an image of a cat. Unfortunately, there was an error in the path or URL, so I could not generate an image. I apologize for not being able to fulfill your request. Is there anything else I can help you with?

Always got error message like this

Traceback (most recent call last):
  File "/JARVIS/server/awesome_chat.py", line 570, in model_inference
    inference_result = huggingface_model_inference(model_id, data, task)
  File "/JARVIS/server/awesome_chat.py", line 411, in huggingface_model_inference
    img_data = image_to_bytes(img_url)
  File "/JARVIS/server/awesome_chat.py", line 213, in image_to_bytes
    load_image(img_url).save(img_byte, format="png")
  File "/anaconda3/envs/jarvis/lib/python3.8/site-packages/diffusers/utils/testing_utils.py", line 268, in load_image
    raise ValueError(
ValueError: Incorrect path or url, URLs must start with `http://` or `https://`, and public/<GENERATED>-1 is not a valid path
2023-04-04 18:18:56,609 - __mp_main__ - WARNING - Inference error: {'message': 'Incorrect path or url, URLs must start with `http://` or `https://`, and public/<GENERATED>-1 is not a valid path'}
2023-04-04 18:18:58,304 - __mp_main__ - WARNING - Inference error: unknown error

or error file path like this

2023-04-04 18:18:58,304 - __mp_main__ - WARNING - Inference error: unknown error
[Errno 2] No such file or directory: 'public/images/1788.png'
Traceback (most recent call last):
  File "/JARVIS/server/awesome_chat.py", line 570, in model_inference
    inference_result = huggingface_model_inference(model_id, data, task)
  File "/JARVIS/server/awesome_chat.py", line 346, in huggingface_model_inference
    img.save(f"public/images/{name}.png")
  File "/anaconda3/envs/jarvis/lib/python3.8/site-packages/PIL/Image.py", line 2429, in save
    fp = builtins.open(filename, "w+b")
FileNotFoundError: [Errno 2] No such file or directory: 'public/images/1788.png'
2023-04-04 18:19:02,531 - __mp_main__ - WARNING - Inference error: {'message': "[Errno 2] No such file or directory: 'public/images/1788.png'"}

Error while starting models_server.py

Hi,
attempting to install this on windows, had a few issues with installing pytorch at the start but after that i got this:

Traceback (most recent call last):
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\modeling_utils.py", line 98, in load_state_dict
    return safetensors.torch.load_file(checkpoint_file, device="cpu")
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\safetensors\torch.py", line 100, in load_file
    result[k] = f.get_tensor(k)
RuntimeError: shape '[640, 2560]' is invalid for input of size 798399

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\modeling_utils.py", line 102, in load_state_dict
    if f.read().startswith("version"):
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 40678: character maps to <undefined>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\AI\JARVIS\server\models_server.py", line 338, in <module>
    pipes = load_pipes(local_deployment)
  File "D:\AI\JARVIS\server\models_server.py", line 315, in load_pipes
    "control": ControlNetModel.from_pretrained(f"{local_fold}/lllyasviel/sd-controlnet-mlsd", torch_dtype=torch.float16),
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\modeling_utils.py", line 557, in from_pretrained
    state_dict = load_state_dict(model_file, variant=variant)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\modeling_utils.py", line 114, in load_state_dict
    raise OSError(
OSError: Unable to load weights from checkpoint file for 'models/lllyasviel/sd-controlnet-mlsd\diffusion_pytorch_model.safetensors' at 'models/lllyasviel/sd-controlnet-mlsd\diffusion_pytorch_model.safetensors'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

any ideas?

Integrate "Segment Anything" from Meta

Is it possible to integrate the functionality of that project?
https://github.com/facebookresearch/segment-anything

add points for readme

Add a project logo: Including a project logo or image can help make the README file more visually appealing and memorable.

1.Add more usage examples: While the existing usage examples are helpful, consider adding more examples to showcase the full capabilities of the project.

Highlight the benefits: Make sure to clearly communicate the benefits and value of using JARVIS. This can help convince potential users or contributors to try the project.

please generate an image where a high-speed train running at high speed.Then please describe the new image with your voice.

Usage with GPT4ALL

Hi,
I don't want to use it with OpenAI but I prefer GPT4ALL. Do you have any recommendations about how to do it?

Could not load model models/ydshieh/vit-gpt2-coco-en

All installation steps has been completed
Running python models_server.py --config config.yaml
Getting this output with error

Fetching 27 files: 100%|██████████████████████████████████████████████████████████████████████████████| 27/27 [00:00<00:00, 338048.38it/s]
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
WARNING:datasets.builder:Found cached dataset cmu-arctic-xvectors (/home/user/.cache/huggingface/datasets/cmu-arctic-xvectors/default/0.0.1/a62fea1f9415e240301ea0042ffad2a3aadf4d1caa7f9a8d9512d631723e781f)
WARNING:datasets.builder:Found cached dataset cmu-arctic-xvectors (/home/user/.cache/huggingface/datasets/cmu-arctic-xvectors/default/0.0.1/a62fea1f9415e240301ea0042ffad2a3aadf4d1caa7f9a8d9512d631723e781f)
Some weights of DPTForDepthEstimation were not initialized from the model checkpoint at models/Intel/dpt-large and are newly initialized: ['neck.fusion_stage.layers.0.residual_layer1.convolution1.weight', 'neck.fusion_stage.layers.0.residual_layer1.convolution2.bias', 'neck.fusion_stage.layers.0.residual_layer1.convolution2.weight', 'neck.fusion_stage.layers.0.residual_layer1.convolution1.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Traceback (most recent call last):
  File "models_server.py", line 342, in <module>
    pipes = load_pipes(local_deployment)
  File "models_server.py", line 261, in load_pipes
    "model": pipeline(task="image-to-text", model=f"{local_fold}/ydshieh/vit-gpt2-coco-en"), 
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/transformers/pipelines/__init__.py", line 779, in pipeline
    framework, model = infer_framework_load_model(
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/transformers/pipelines/base.py", line 271, in infer_framework_load_model
    raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
ValueError: Could not load model models/ydshieh/vit-gpt2-coco-en with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForVision2Seq'>, <class 'transformers.models.vision_encoder_decoder.modeling_vision_encoder_decoder.VisionEncoderDecoderModel'>).

Colab deployment insuffient hardware space

Hi there,
Love this idea and would love to try and contribute if possible.
Now I'm deploying it on Colab but failed a few times with the error "not enough hardware space".
I'm seeing there're close to 200G space but still running out of space in the end which is not consistant with the description of 80G required for full local model.

Please let me know if you have any questions.
Thanks,

Mavel trademark issue

Description

Hello Microsoft,
Congratulations on turning the project into working code, looking forward to contributing to this project. This architecture with enough training can possibly be the way to AGI.

There are a few concerns however. The name JARVIS appears to be a registered trademark under Marvel Entertainment Studios under Class 9 for computer application software for use as personal digital assistants. [reference. 1]. As well as Blackberry has a history of similar trademark under their digital assistant application categories.

Proposed Changes

Attain Copyright or Settle for a trademark usage

Where is the code?

KeyError choices in gradio demo

2023-04-05 20:33:46,053 - awesome_chat - INFO - input:
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/routes.py", line 393, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 1108, in process_api
    result = await self.call_function(
  File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 915, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "run_gradio_demo.py", line 81, in bot
    message = chat_huggingface(all_messages, OPENAI_KEY)
  File "/home/ec2-user/SageMaker/JARVIS/server/awesome_chat.py", line 852, in chat_huggingface
    task_str = parse_task(context, input, openaikey).strip()
  File "/home/ec2-user/SageMaker/JARVIS/server/awesome_chat.py", line 306, in parse_task
    return send_request(data)
  File "/home/ec2-user/SageMaker/JARVIS/server/awesome_chat.py", line 173, in send_request
    return response.json()["choices"][0]["text"].strip()
KeyError: 'choices'

models_server.py returns error

(jarvis) [root@myflink server]# python models_server.py --config config.yaml
Traceback (most recent call last):
File "models_server.py", line 10, in
from diffusers.utils import export_to_video
ImportError: cannot import name 'export_to_video' from 'diffusers.utils' (/root/anaconda3/envs/jarvis/lib/python3.8/site-packages/diffusers/utils/init.py)

awesome_chat.py returns error about openai api key

I have quadruple checked my openai api key is correct. I copied and pasted it to avoid typo, I even generated a new key after many attempts/fails. However, no matter what key I generate and write to config.yaml, when I run awesome_chat.py it insists that I have an incorrect openai key.

I also tried running npm run dev and the browser opens and I see the HuggingGPT page and text box, but anytime I submit any prompt, system returns "Something seems seems wrong". Interestingly, localhost is opened to port 9999 when running npm run dev, but in config.yaml it has localhost port set to 8003.

What is it I am doing wrong? I feel like I am so close, but so far. :)

Jarvis let's make one old but good idea for smart home

Dynamically operate a browser - for iterating between testing / updating code

Hey, so I have a game built from HTML/Javascript. I would like to see how well JARVIS could try to understand and then modify the existing game. Make changes... re-run the game... play it for finding bugs, etc.

For this it will need to learn how to press buttons, type text into fields, and

View the existing page with vision
And/or
Parse the page, over-and-over again, as it takes its actions, to understand what is on the page.

How might this be achieved? What hugging face AI's would I have to pull in?

Add a contributing.md file

Why isn't there a contributing.md file to help give guidance on how or even if we can contribute to this project?

I think adding one would help give guidance on how or if we can contribute to this project.

So Mac can not use this?

Macs are not using NVIDIA display card, so Mac can not use this right?

Would it support Azure OpenAI?

Hi, I only saw OpenAI API key in config.yml. Does it support Azure OpenAI? Or is there any plans to support Azure OpenAI?

Gradio. Built-in example - Cat Video. FileNotFoundError: [Errno 2] No such file or directory

Server is up an running.
Commit SHA - bc66e5a
Using gradio python run_gradio_demo.py --config config.gradio.yaml:

Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

Running built-in example based on the /examples/a.jpg, please generate a video and audio . Gradio terminal output:

2023-04-06 16:33:43,346 - awesome_chat - INFO - ********************************************************************************
2023-04-06 16:33:43,352 - awesome_chat - INFO - input: based on the /examples/a.jpg, please generate a video and audio
2023-04-06 16:33:47,719 - awesome_chat - INFO - [{"task": "image-to-text", "id": 0, "dep": [-1], "args": {"image": "/examples/a.jpg" }}, {"task": "text-to-video", "id": 1, "dep": [0], "args": {"text": "<GENERATED>-0" }}, {"task": "text-to-speech", "id": 2, "dep": [0], "args": {"text": "<GENERATED>-0" }}]
2023-04-06 16:34:36,874 - awesome_chat - INFO - response: I have carefully considered your request and I have generated a video and audio based on the image you provided. The workflow I used is as follows: First, I used the image-to-text model 'nlpconnect/vit-gpt2-image-captioning' to generate the text 'a cat sitting on a window sill looking out'. Then, I used the text-to-video model 'damo-vilab/text-to-video-ms-1.7b' to generate the video '/videos/293f.mp4'. Finally, I used the text-to-speech model 'facebook/fastspeech2-en-ljspeech' to generate the audio '/audios/19fc.flac'. The complete path of the video and audio are '/videos/293f.mp4' and '/audios/19fc.flac' respectively. I hope this answer your request. Is there anything else I can help you with?
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/routes.py", line 393, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 1111, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 1045, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/components.py", line 4333, in postprocess
    self._postprocess_chat_messages(message_pair[1]),
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/components.py", line 4296, in _postprocess_chat_messages
    filepath = self.make_temp_copy_if_needed(filepath)
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/components.py", line 245, in make_temp_copy_if_needed
    temp_dir = self.hash_file(file_path)
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/components.py", line 217, in hash_file
    with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'public//videos/293f.mp4'

Cross-Origin Request Blocked

Clean install. The server is running.

Ubuntu 22.04
Firefox/Chrome

cd web/
npm run dev

Result of submitting any prompt:

With the following in the console:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8004/hugginggpt. (Reason: CORS request did not succeed). Status code: (null).

must i have a openai key for this project?

hi,dear
as the title,if i don't have a openai key ,could i use the rp ?
thx

"Encountered 8 files may not copied correctly on windows"

Have this when downloading models.
And I think models disk usage should also be listed on system requirements.

Solving environment: failed with initial frozen solve.

Environments

OS: macOS 11.5.1 (20G80)
Hardware: MacBook Pro (Retina, 15-inch, Mid 2015)

conda info

     active environment : jarvis
    active env location : /usr/local/anaconda3/envs/jarvis
            shell level : 1
       user config file : /Users/bruce/.condarc
 populated config files :
          conda version : 23.1.0
    conda-build version : 3.23.3
         python version : 3.10.9.final.0
       virtual packages : __archspec=1=x86_64
                          __osx=10.16=0
                          __unix=0=0
       base environment : /usr/local/anaconda3  (writable)
      conda av data dir : /usr/local/anaconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /usr/local/anaconda3/pkgs
                          /Users/bruce/.conda/pkgs
       envs directories : /usr/local/anaconda3/envs
                          /Users/bruce/.conda/envs
               platform : osx-64
             user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.9 Darwin/20.6.0 OSX/10.16
                UID:GID : 501:20
             netrc file : None
           offline mode : False

conda config --show-sources

empty, no any config

conda list --show-channel-urls

# packages in environment at /usr/local/anaconda3/envs/jarvis:
#
# Name                    Version                   Build  Channel
ca-certificates           2023.01.10           hecd8cb5_0    defaults
certifi                   2022.12.7        py38hecd8cb5_0    defaults
libcxx                    14.0.6               h9765a3e_0    defaults
libffi                    3.4.2                hecd8cb5_6    defaults
ncurses                   6.4                  hcec6c5f_0    defaults
openssl                   1.1.1t               hca72f7f_0    defaults
pip                       23.0.1           py38hecd8cb5_0    defaults
python                    3.8.16               h218abb5_3    defaults
readline                  8.2                  hca72f7f_0    defaults
setuptools                65.6.3           py38hecd8cb5_0    defaults
sqlite                    3.41.1               h6c40b1e_0    defaults
tk                        8.6.12               h5d9f67b_0    defaults
wheel                     0.38.4           py38hecd8cb5_0    defaults
xz                        5.2.10               h6c40b1e_1    defaults
zlib                      1.2.13               h4dc903c_0    defaults

Errors

Run the following command

conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

Then it shows the following error

Erros

> conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: -
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
Examining conflict for torchvision pytorch:  38%|███████████████                         | 3/8 [00:04<00:07,  1.48s//failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - torchaudio -> python[version='>=2.7,<2.8.0a0|>=3.11,<3.12.0a0']

Your python: python=3.8

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package setuptools conflicts for:
pytorch -> jinja2 -> setuptools
torchvision -> setuptools
python=3.8 -> pip -> setuptools

Package pytorch conflicts for:
torchaudio -> pytorch[version='1.10.0|1.10.1|1.10.2|1.11.0|1.12.0|1.12.1|1.13.0|1.13.1|2.0.0|1.9.1|1.9.0|1.8.1|1.8.0|1.7.1|1.7.0|1.6.0|1.5.1']
torchvision -> pytorch[version='1.10.0|1.10.1|1.10.2|1.11.0|1.12.0|1.12.1|1.13.0|1.13.1|2.0.0|1.9.1|1.9.0|1.8.1|1.8.0|1.7.1|1.7.0|1.6.0|1.5.1|1.7.1.*|1.3.1.*']

Package six conflicts for:
pytorch -> mkl-service[version='>=2.3.0,<3.0a0'] -> six
torchvision -> six

Package libcxxabi conflicts for:
pytorch -> libcxx[version='>=4.0.1'] -> libcxxabi==4.0.1[build='hebd6815_0|hcfea43d_1']
python=3.8 -> libcxx[version='>=4.0.1'] -> libcxxabi==4.0.1[build='hebd6815_0|hcfea43d_1']

Can Nvidia's 40 series graphics card be used for the current project?

as I can see that the System Requirements is as follows:

Ubuntu 16.04 LTS
NVIDIA GeForce RTX 3090 * 1
RAM > 24GB

Whisper-base model cannot be loaded

Hi guys, when I'm running python models_server.py --config config.yaml I get this error:

raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
ValueError: Could not load model models/openai/whisper-base with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCTC'>, <class 'transformers.models.auto.modeling_auto.AutoModelForSpeechSeq2Seq'>, <class 'transformers.models.whisper.modeling_whisper.WhisperForConditionalGeneration'>).

使用stanford_alpaca去替代GPT-3.5turbo

stanford_alpaca是开源模型，可以保证在openai服务不可用的地方依然可以使用到HuggingGPT

Gradio. Built-in example - Zebras. TypeError: expected string or bytes-like object

Server is up an running. Using gradio python run_gradio_demo.py --config config.gradio.yaml:

Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

Running built-in example Given a collection of image A: /examples/a.jpg, B: /examples/b.jpg, C: /examples/c.jpg, please tell me how many zebras in these picture?. Gradio terminal output:

2023-04-06 15:41:17,757 - awesome_chat - INFO - ********************************************************************************
2023-04-06 15:41:17,758 - awesome_chat - INFO - input: Given a collection of image A: /examples/a.jpg, B: /examples/b.jpg, C: /examples/c.jpg, please tell me how many zebras in these picture?
2023-04-06 15:41:30,058 - awesome_chat - INFO - [{"task": "image-to-text", "id": 0, "dep": [-1], "args": {"image": "/examples/a.jpg" }}, {"task": "object-detection", "id": 1, "dep": [-1], "args": {"image": "/examples/a.jpg" }}, {"task": "visual-question-answering", "id": 2, "dep": [1], "args": {"image": "<GENERATED>-1", "text": "How many zebras in the picture?" }}, {"task": "image-to-text", "id": 3, "dep": [-1], "args": {"image": "/examples/b.jpg" }}, {"task": "object-detection", "id": 4, "dep": [-1], "args": {"image": "/examples/b.jpg" }}, {"task": "visual-question-answering", "id": 5, "dep": [4], "args": {"image": "<GENERATED>-4", "text": "How many zebras in the picture?" }}, {"task": "image-to-text", "id": 6, "dep": [-1], "args": {"image": "/examples/c.jpg" }}, {"task": "object-detection", "id": 7, "dep": [-1], "args": {"image": "/examples/c.jpg" }}, {"task": "visual-question-answering", "id": 8, "dep": [7], "args": {"image": "<GENERATED>-7", "text": "How many zebras in the picture?" }}]
2023-04-06 15:41:50,233 - awesome_chat - INFO - response: Based on the inference results, there are two zebras in the picture. 

My workflow for your request is as follows: 

1. I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate a text description for each image. 

2. Then I used the object-detection model facebook/detr-resnet-50 to detect the objects in the image and generate an image with predicted boxes. 

3. Finally, I used the visual-question-answering model dandelin/vilt-b32-finetuned-vqa to answer your question. 

For the image A: /examples/a.jpg, the object-detection model detected a cat and a potted plant in the image. The visual-question-answering model predicted that there are 0 zebras in the picture. 

For the image B: /examples/b.jpg, the object-detection model detected a zebra in the image. The visual-question-answering model predicted that there is 1 zebra in the picture. 

For the image C: /examples/c.jpg, the object-detection model detected three zebras in the image. The visual-question-answering model predicted that there are 2 zebras in the picture. 

Therefore, there are two zebras in the picture.
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/routes.py", line 393, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 1108, in process_api
    result = await self.call_function(
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 915, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "run_gradio_demo.py", line 82, in bot
    image_urls, audio_urls, video_urls = extract_medias(message)
  File "run_gradio_demo.py", line 18, in extract_medias
    for match in image_pattern.finditer(message):
TypeError: expected string or bytes-like object

Same error is thrown in the following built-in examples:

Please generate a canny image based on /examples/f.jpg
what is in the examples/a.jpg
generate a video and audio about a dog is running on the grass
based on the /examples/a.jpg, please generate a video and audio
based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image

I get the following error:

Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/jarvis/lib/python3.8/site-packages/transformers/modeling_utils.py", line 415, in load_state_dict
    return torch.load(checkpoint_file, map_location="cpu")
  File "/home/ubuntu/miniconda3/envs/jarvis/lib/python3.8/site-packages/torch/serialization.py", line 795, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/ubuntu/miniconda3/envs/jarvis/lib/python3.8/site-packages/torch/serialization.py", line 1002, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.

suggestion to requirement.txt install failure

pip install controlnet-aux==0.0.1

requirement.txt 前3项如果出现安装问题，可以下载后，copy到javis环境目录
第三个，直接使用上面的pip安装。