bug: Model downloaded from Hugging Face can only run on CPU because it doesn't retrieve the correct GGUF model's metadata about jan HOT 6 CLOSED

In my case, on Linux, trying to use the GPU I see on the logs:

ERROR Could not load engine: Could not load library "/home/jose/.config/Jan/data/extensions/@janhq/inference-cortex-extension/dist/bin/linux-cuda-12-0/engines/cortex.llamacpp/libengine.so"
libcudart.so.12: cannot open shared object file: No such file or directory - server.cc:299

@milen-prg On what previous version it worked for you?

from jan.

@josepgl , on v0.5.3 worked (there was another problem, the models was disappeared, but here easy helped me then).
On Windows where to see the logs?

from jan.

@milen-prg in Settings > Advanced Settings > Jan Data Folder is the data path, I see a log folder there in Linux.

from jan.

There is logs folder, but it is empty.

from jan.

There is logs folder, but it is empty.

We only store your logs for 24h

from jan.

This is a known issue. The Hugging Face model download from the Search Box in model hub is pretty broken for now, and it doesn't retrieve the correct GGUF model's metadata. We've filed an issue on this and are working on the fix from the engine: #3558

In the meantime, please help us add an ngl setting to the settings section for now to enable GPU acceleration. It worked fine before because the previous versions hardcoded an ngl setting, which is hacky and not correct for all models.

from jan.