https://huggingface.co/TheBloke/koala-7B-GGML/resolve/main/koala-7B.ggml.q4_0.bin
this is not available getting 404

https://huggingface.co/TheBloke/koala-7B-GGML/resolve/main/koala-7B.ggmlv3.q4_0.bin installed this version but getting error while running

Run the chatbot

!python3 privateGPT.py

Using embedded DuckDB with persistence: data will be stored in: db
CUDA error 35 at /content/llama-cpp-python/vendor/llama.cpp/ggml-cuda.cu:2106: CUDA driver version is insufficient for CUDA runtime version

what is an issue?

Hi, I am trying to run this colab and gotten the error below when trying the run the chatbot. Do you have any idea on the cause?

# Run the chatbot !python3 privateGPT.py

Using embedded DuckDB with persistence: data will be stored in: db
ggml_init_cublas: found 1 CUDA devices:
Device 0: Tesla T4
Traceback (most recent call last):
File "/content/privateGPT/privateGPT.py", line 101, in
main()
File "/content/privateGPT/privateGPT.py", line 56, in main
llm = LlamaCpp(model_path=model_path, n_ctx=model_n_ctx, callbacks=callbacks, verbose=False, n_gpu_layers=calculate_layer_count())
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCpp
root
Could not load Llama model from path: models/koala-7B.ggml.q4_0.bin. Received error Model path does not exist: models/koala-7B.ggml.q4_0.bin (type=value_error)
Exception ignored in: <function Llama.del at 0x7f7f331283a0>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/llama_cpp_python-0.1.59-py3.10-linux-x86_64.egg/llama_cpp/llama.py", line 1424, in del
if self.ctx is not None:
AttributeError: 'Llama' object has no attribute 'ctx'

ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090
llama.cpp: loading model from models/koala-7B.ggmlv3.q4_0.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 1000
llama_model_load_internal: n_embd = 4096
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 32
llama_model_load_internal: n_layer = 32
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 11008
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 0.07 MB
llama_model_load_internal: using CUDA for GPU acceleration
llama_model_load_internal: mem required = 5407.71 MB (+ 1026.00 MB per state)
llama_model_load_internal: offloading 0 repeating layers to GPU
llama_model_load_internal: offloaded 0/35 layers to GPU
llama_model_load_internal: total VRAM used: 512 MB

when i run the script, everything works fine.
except when i request my question.

# Run the chatbot
!python3 privateGPT.py

it gives :

Using embedded DuckDB with persistence: data will be stored in: db
ggml_init_cublas: found 1 CUDA devices:
  Device 0: Tesla T4, compute capability 7.5
llama.cpp: loading model from models/koala-7B.ggmlv3.q5_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 1000
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_head_kv  = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 8 (mostly Q5_0)
llama_model_load_internal: model size = 7B
who is Kafka ?
LLAMA_ASSERT: /content/llama-cpp-python/vendor/llama.cpp/llama.cpp:1800: !!kv_self.ctx

it display this line : LLAMA_ASSERT: /content/llama-cpp-python/vendor/llama.cpp/llama.cpp:1800: !!kv_self.ctx without any answer.
i guest it crashes on qa(query).

Hi, can you help me?

I have issue when run chatbot :

%cd /content/privateGPT/
!python3 privateGPT.py

and get errors:

".....
root
Could not load Llama model from path : models/koadla-7B.ggmlv3.q4_0.bin ...."

Thank you in advance

Hi, I run the privateGPT.py successfully and could get into "Enter a Query:".

I managed to type in the prompt but it doesn't give me any completion nor error, it just keeps loading for long time. Do you have any clue?

This is my value in .env
!sed -i 's/n_gpu_layers=10' /content/privateGPT/.env
!sed -i 's/MODEL_TYPE=GPT4All' /content/privateGPT/.env
!sed -i 's/ggml-gpt4all-j-v1.3-groovy.bin' /content/privateGPT/.env

This is my GPU setting in Colab.