containers / ai-lab-recipes Goto Github PK

@MichaelClifford i see a lot of calls fialing wiht 404. Is it expected?

This would lead to:

Error: no Containerfile or Dockerfile specified or found in context directory, C:\Users\Jeff\work\src\github.com\containers\ai-lab-recipes\model_servers\whispercpp: Le fichier spécifié est introuvable.

It's not clear how to utilize GPU with recipes, if GPU is available - should add documentation for each app where this is possible.

The script is ask not summarize

https://github.com/MichaelClifford/locallm/blob/8a3095328385fb1cc49ec87898dbf44e9018ab54/chatbot/README.md?plain=1#L62

Some of the recipe READMEs explain how to build bootable container images (e.g., https://github.com/containers/ai-lab-recipes/tree/main/recipes/natural_language_processing/chatbot#embed-the-ai-application-in-a-bootable-container-image).

To complete the e2e UX, I want the docs/READMEs to also elaborate on how to build disk images. I want to cover CLI use cases and also point to Podman Desktop Bootc Extension for desktop users.

Cc: @rhatdan @sallyom

Following the README instructions on mac, I ran:

make -f Makefile build && make -f Makefile run

build step completes. And the it runs:

cd ../../models && \
        podman run -it -d -p 8001:8001 -v ./mistral-7b-instruct-v0.1.Q4_K_M.gguf:/locallm/models/mistral-7b-instruct-v0.1.Q4_K_M.gguf:ro -e MODEL_PATH=/locallm/models/mistral-7b-instruct-v0.1.Q4_K_M.gguf -e HOST=0.0.0.0 -e PORT=8001 quay.io/ai-lab/model_servers/llamacpp_python:latest

The container instantly crashes with:

llama_model_load: error loading model: failed to open /locallm/models/mistral-7b-instruct-v0.1.Q4_K_M.gguf: Permission denied
llama_load_model_from_file: failed to load model
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/__main__.py", line 88, in <module>
    main()
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/__main__.py", line 74, in main
    app = create_app(
          ^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/app.py", line 138, in create_app
    set_llama_proxy(model_settings=model_settings)
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/app.py", line 75, in set_llama_proxy
    _llama_proxy = LlamaProxy(models=model_settings)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/model.py", line 31, in __init__
    self._current_model = self.load_llama_from_model_settings(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/model.py", line 138, in load_llama_from_model_settings
    _model = create_fn(
             ^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama.py", line 314, in __init__
    self._model = _LlamaModel(
                  ^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/_internals.py", line 55, in __init__
    raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: /locallm/models/mistral-7b-instruct-v0.1.Q4_K_M.gguf

I think that the it might be an issue with the BIND_MOUNT_OPTION in the model_severs common Makefile.

Both chat_langchain and rag_langchain do not have memory components. Meaning, that you cannot ask follow up questions during a chat session.

We should add the a chat memory component that allows for asking follow up questions. It should also gracefully handle the case in which the chat history exceeds the models context window.

This PR removed the model-converter image, it needs to be added back when we figure out why it is failing.

path is incorrect as the context dir is audio_to_text

See

Caused by 7677970

See

The current repo mirror strategy to drive builds down is not scaleable. We should look to move to using self-hosted Github runners where we can mount the models, stored on persistent storage, to the filesystem in such a way that our tests will not run out storage, and will not have flakiness due to multi-gigabyte model downloads. Even if we could limp along with our current solution, swapping to this strategy will be a requirement of testing our multi-model feature in llamacpp_python model_server.

Initial idea was discussed in the thread beginning with: https://redhat-internal.slack.com/archives/C06S75ZF9JT/p1713089733094399?thread_ts=1712828397.645709&cid=C06S75ZF9JT .

We plan to implement this after Release 1.0 so as to not interfere, but the POC can be developed and run alongside our workloads leading up to and during release.

/assign @lmilbaum
/assign @Gregory-Pereira

Once #63 is complete and we have an API for the whisper model service. We should add an app that allows a user to upload an audio file and get text response.

Ideally, we'd like to be able to send the output of this model to the summarizer or rag app to interact with it further.

Here is the error in the client application:

We should add a tool into the whisper_playground workflow that does this file conversion for the user.

ffmpeg -i <input.mp3> -ar 16000 -ac 1 -c:a pcm_s16le <output.wav>

As raised by @lstocchi in containers/podman-desktop-extension-ai-lab#869 (review) user may try to use some format which are not compatible.

As ffmpeg is available, the whisper_client.py could check for the format of the source audio and convert it to the expected format before sending it.

cc @MichaelClifford

The following images would be interesting to have available to a registry

• models/convert_models

Required by containers/podman-desktop-extension-ai-lab#542

• model_servers/llamacpp_python/cuda
• model_servers/llamacpp_python/vulkan

The docs mention XDG_RUNTIME_DIR for mounting the auth.json file:

podman build --build-arg "sshpubkey=$(cat ~/.ssh/id_rsa.pub)" \
           --security-opt label=disable \
	   -v ${XDG_RUNTIME_DIR}/containers/auth.json:/run/containers/0/auth.json \
	   --cap-add SYS_ADMIN \
	   -t quay.io/yourrepo/youros:tag .

The docs further mention building on the Mac. But, on the Mac the file resides in $HOME/.config/containers/auth.json.

I think the docs need some tweaking to be portable across Linux/Mac/Win.

add quadlet

We've added convenient make targets & we need to document them!

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(deps): update dependency safetensors to v0.4.4

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

• chore(deps): update all dependencies (Pygments, attrs, aws-actions/configure-aws-credentials, blinker, cachetools, certifi, containers.podman, fastapi, huggingface-hub, jsonschema, langchain, mwader/static-ffmpeg, numpy, nvcr.io/nvidia/cuda, packaging, pillow, pip, protobuf, pyarrow, pydantic, pydantic_core, pydeck, pypdf, pytest, quay.io/containers/podman, referencing, regex, requests, rpds-py, selenium, sentence-transformers, starlette, streamlit, sympy, tenacity, torch, torchvision, transformers, typing_extensions)

Detected dependencies

The AI Lab extension displays the descriptions of the recipes which at the present are quite repetitive:

• They all start with "This is a[n] ..."
• All claim it's a "demo application" which doesn't sound fit for production
• I'd prefer a short description of the use cases. For instance, "Summarizer: Summarize text files in a web front end." (or something similar).

We will be releasing version 1.0 to matchup with Podman Desktop AI Lab.
Release Date: Wednesday, March 17th

Release Criteria:

• Brief change log or summary of issues tackled and new features
  • (OPTIONAL): A comparison of where we are in relation to the Podman Desktop team
• A commit is made to trigger every workflow in the repo with tests passing
• Potentially switch to granite mode by default, if it is released?

...
Thoughts @lmilbaum @sallyom @MichaelClifford @rhatdan ?

See

Also

We received user feedback that the recipe descriptions can be confusing. Let's take the ChatBot as an example (see screenshot) below. The description does not target the desktop use case but the terminal/CLI one. So there is some friction.

Since the AI Lab Recipes target both use cases, I guess that the READMEs should first target the desktop use case and then the CLI use with clear descriptions of the target audience?

Happens when the summarizer image is run

Need to add functional testing for:

(pending #247)

• summarizer
• codegen
• rag
• whisper model-server
• llamacpp-python-cuda
• llamacpp-python-vulkan
• audio-to-text

The object_detection_server starts with the following error:

Traceback (most recent call last):
  File "/opt/app-root/bin/uvicorn", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/uvicorn/main.py", line 409, in main
    run(
  File "/opt/app-root/lib64/python3.11/site-packages/uvicorn/main.py", line 575, in run
    server.run()
  File "/opt/app-root/lib64/python3.11/site-packages/uvicorn/server.py", line 65, in run
    return asyncio.run(self.serve(sockets=sockets))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/uvicorn/server.py", line 69, in serve
    await self._serve(sockets)
  File "/opt/app-root/lib64/python3.11/site-packages/uvicorn/server.py", line 76, in _serve
    config.load()
  File "/opt/app-root/lib64/python3.11/site-packages/uvicorn/config.py", line 433, in load
    self.loaded_app = import_from_string(self.app)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/uvicorn/importer.py", line 19, in import_from_string
    module = importlib.import_module(module_str)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/locallm/object_detection_server.py", line 15, in <module>
    processor = AutoImageProcessor.from_pretrained(model, revision=revision)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�  File "/opt/app-root/lib64/python3.11/site-packages/transformers/models/auto/image_processing_auto.py", line 358, in from_pretrained
    config_dict, _ = ImageProcessingMixin.get_image_processor_dict(pretrained_model_name_or_path, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�  File "/opt/app-root/lib64/python3.11/site-packages/transformers/image_processing_utils.py", line 363, in get_image_processor_dict
    text = reader.read()
           ^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte

Now that we have a whispercpp image that can be used to convert audio to text, want to use it the same way we use the existing playground image, as a model service with an exposed API. Unlike Llama_cpp_python, whisper.cpp does not appear to come with a prebuilt api server for the model. So we should create a light weight API to use for this model type.

That would be a great plus for people demoing the audio_to_text recipe

Some of them use - others use _

re https://github.com/MichaelClifford/locallm/blob/a82ac930d2905ad390a0fd3fcb3ade5a2319bde7/Containerfile-rh#L6-L7

First there's no reason to install wget - we already have curl.

But even better, I think we want to use COPY --link here, as is recommended by e.g. https://github.com/depot/depot.ai#usage

With the latest changes on the recipes, the readme are not providing the same level of details.

Now: https://github.com/redhat-et/locallm/blob/main/summarizer-langchain/README.md
Before: https://github.com/redhat-et/locallm/blob/5ff3767ff3e265a917a4f604211557d2511d41b4/summarizer/README.md

We would like to get back to the previous version of the content, aligned with the latest changes.

System information

• Windows 11
• WSL version: 2.1.5.0

Details

To access the GPU in a container while using podman on windows, we have to make some commands¹

In the following example, we are using official nvidia images. Not images provided by this repository

podman run \
    --device=/dev/dxg \
    --mount type=bind,source=/usr/lib/wsl,target=/usr/lib/wsl \
    --gpus all \
    --entrypoint=sh \
    docker.io/nvidia/cuda:12.3.1-devel-ubuntu22.04 \
    -c '/usr/bin/ln -s /usr/lib/wsl/lib/* /usr/lib/x86_64-linux-gnu/ && PATH="${PATH}:/usr/lib/wsl/lib/" && nvidia-smi'

This will output the following on my system

Fri Mar 22 17:42:48 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.53       Driver Version: 497.29       CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:08:00.0  On |                  N/A |
| 34%   38C    P0    28W / 120W |   1344MiB /  6144MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Using model_servers/llamacpp/cuda/Containerfile

In the following, I will try to reproduce the GPU access I had using official nvidia images with the one from this repository

Reproduce

git clone https://github.com/redhat-et/locallm
cd locallm\model_servers\llamacpp\
podman build --platform linux/amd64 -t chatbot:service-cuda -f cuda/Containerfile .

Once I have the chatbot:service-cuda image, I will run the same scenario I made with the previous ubuntu based image

podman run \
    --device=/dev/dxg \
    --mount type=bind,source=/usr/lib/wsl,target=/usr/lib/wsl \
    --gpus all \
    --entrypoint=sh \
    localhost/chatbot:service-cuda \
    -c '/usr/bin/ln -s /usr/lib/wsl/lib/* /usr/lib/x86_64-linux-gnu/ && PATH="${PATH}:/usr/lib/wsl/lib/" && nvidia-smi'

but this operation will result in the following

ln: target '/usr/lib/x86_64-linux-gnu/': No such file or directory

STEP 11/11: RUN podman system reset --force 2>/dev/null
Error: building at STEP "RUN podman system reset --force 2>/dev/null": while running runtime: exit status 125

Looking at the containerfile I'm not sure we need this line. It fails for me regardless of ARCH=aarch64 or x86_64. It seems to work when I drop it. I'm testing with make FROM=registry.redhat.io/rhel9-beta/rhel-bootc:9.4 CONTAINERFILE=Containerfile.nocache bootc with the summarizer app

I would like to change the structure of this repo a bit to make it more intuitive and easier to maintain as it continues to grow. Below please find a sketch of the proposed structure.

The main changes would be:

• playground/ would be moved into model_servers/ and renamed llamacpp_python/
• A new directory called recipes/ with N subdirectories for each recipe category. Right now that is natural_language_processing/, computer_vision/, audio/ and multimodal/.
• All existing recipes will be moved to the correct subdirectory.
• A new directory called VectorDBs/. With the intent to add any new component types that may arise in the future in the same way to the root directory.
• Move chromadb/ into VectorDBs/

Please let me know what you all think of the proposed changes and how they might affect any dependencies on this repo.

locallm:
|-- assets
    |-- image.1.png
    |-- image2.png
    |-- ...
|-- data
    |-- file.txt
    |-- ...
|-- models
    |-- Model1Folder
    |-- Model2Folder
    |-- Convert_Models
    |-- ...
|-- model_servers
    |-- llamacpp_python
        |-- base
        |-- cuda
        |-- vulkan
        |-- ....
    |-- ollama
        |-- base
        |-- ....
    |-- caikit
        |-- base
        |-- ....
    |-- WhisperCPP
        |-- base
        |-- ....
    |-- ...
|-- recipes
    |-- Natural Language
        |-- Chat
        |-- Summary
        |-- RAG
        |-- Fine Tune
        |-- ...
    |-- Computer Vision
        |-- Object Detection
        |-- ...
    |-- Audio
        |-- Transcription
        |-- ...
    |-- Multimodal
        |-- Image Description
        |-- ...
|-- VectorDBs
    |-- ChromaDB
    |-- ...

Need to add make bootc testing for:
(pending #254)

• codegen
• summarizer
• rag
• audio-to-text

Currently mistral is containerized in this workflow , but we want to add these to build with the model-servers, and to be promoted to ai-lab with the promotion workflow from ghcr to quay.io/ai-lab

A recipe that does the following:

Given a small set of json schemas (Ok to start with 1), create an LLM application where a user inputs their responses to the fields of the json schema using unstructured natural language and the model returns a correctly populated json.

A simple example to illustrate how this should work would be placing an order at a fast food restaurant. The user simply states their order, and the LLM generates the appropriate json that can be submitted to their oder management system.

whisper images should be added to GH workflow and need to create quadlet files

Add links to the repos (make files) in each of the AI recipe readme

Discussed during the Weekly Demo on Apr 15. It's challenging for the user to connect with the AI recipe on Podman Desktop to convert the images with bootc utilizing the make files, including the discoverability of the make files so adding the links to repos with the make files in the readme(s) should be helpful

Should be MODEL_ENDPOINT

llama-cpp detects CPU features like AVX, AVX2, FMA3, and F16C at build time. If the container is built on a machine that supports these instruction sets, then the binary won't work on CPUs without these instructions.

References:

• https://github.com/ggerganov/llama.cpp/blob/d26e8b669dbf1f5f5a0afe4d2d885e86cf566302/CMakeLists.txt#L73-L78
• abetlen/llama-cpp-python#412

Credits to @bbrowning for figuring this out. He suggested to build llama-cpp-python with CMAKE_ARGS="-DLLAMA_CUBLAS=on -DLLAMA_AVX2=OFF -DLLAMA_FMA=OFF -DLLAMA_F16C=OFF"

/cc @MichaelClifford

Initial idea: Deploy the chatbot LLM to the ET cluster so we can start using our own tools and ideating on how we test and evaluate model performance.

Spinoff Idea: Might be cool to also do this with the RAG application once its ready so we can evaluate RAG cloud vs edge models and how we reconcile / update the 2 with data

Stretch goal: It would be cool if we can deploy chatbot to review readme changes in this repo.

Currently, the multi-arch builds for RAG-langchain images are broken, only arm64 images are being built. Need to add automation for x86 images.

Based on how similar the recipe workflows are we could refactor them into a singular workflow with individual job for any recipe that requires steps outside the standard.

Codegen recipe should include a more code focused use case. In addition to the chat interface, we should demostreate how you could use this in place of copilot in Vscode.

See the audio_to_text as an example

• Research a few tools + methods for implementing guard rails for LLM applications.
• Add a recipe that provides developers with an example of implementing guard rails.