Welcome to GGUF'n'Go, your go-to tool for easily creating quantized versions of Hugging Face models in the GGUF format. Whether you're a machine learning enthusiast or a professional looking to optimize your models, GGUF'n'Go simplifies the process of converting models into various quantized formats for efficient deployment.

Quantization is a key technique in reducing the size of large language models (LLMs) without significantly compromising their performance. By converting models to lower precision formats, you can save on storage, reduce latency, and enable deployment on resource-constrained environments. GGUF'n'Go supports a wide range of quantization types via Llama.cpp, ensuring flexibility and efficiency for your specific needs.

Ensure that you have curl and unzip installed on your system. These tools are essential for downloading and extracting the necessary files.

Follow these steps to set up the project:

1. Clone the repository:

   git clone [email protected]:thesven/GGUF-n-Go.git
   cd GGUF-n-Go

2. Make the setup script executable and run it. This will download and compile llama.cpp, as well as download the wiki.train.raw dataset.

   chmod +x setup.sh
   ./setup.sh

1. Copy the example configuration file and customize it with the Hugging Face model you wish to convert to the GGUF format:

   cp gguf.example.toml gguf.toml

2. Edit gguf.toml to specify your model details.

Execute the conversion process with the following command:

python ./gguf_n_go.py --config ./gguf.toml

This will generate the quantized model in the specified formats.

GGUF'n'Go supports a comprehensive range of quantization types, each offering different trade-offs between model size, performance, and quality. Please see the list of quantization types in the gguf.example.toml file. If you notice any missing, please feel free to add them.

Feel free to fork this repository and submit pull requests to contribute to the project. All ideas and suggestions are welcome!