Hugging Face has become synonymous with state-of-the-art machine learning models, particularly in natural language processing. On the other hand, the GGUF file format, though less well-known, serves specific purposes that necessitate the conversion of models into this format.

This article provides a comprehensive walkthrough on how to convert any Hugging Face model to GGUF, ensuring your models are versatile across different platforms and applications.

What is the GGUF File Format?

The GGUF file format is a specialized file type used in certain machine learning and data processing environments. It is designed to encapsulate model data and configuration in a manner that is optimized for quick loading and high efficiency in specific scenarios or platforms. Understanding how to convert to this format can be crucial for deploying Hugging Face models in environments that require GGUF compatibility.

Steps to convert any huggingface model to gguf file format

Converting a Hugging Face model to the GGUF (Georgi Gerganov’s Universal Format) file format involves a series of steps that leverage tools from the Hugging Face Hub and the Llama.cpp library. This conversion process facilitates the deployment of models on local systems or in environments where efficiency and speed are critical.

Here’s a step-by-step guide on how to achieve this:

Step 1: Download the Hugging Face Model

There are two main methods for downloading a Hugging Face model. You can use the Hugging Face Hub, a repository for various machine learning models, or the Transformers library, a Python library for working with Hugging Face models.

We use the `huggingface_hub` library to download the model. For example, we are converting “microsoft/Phi-3-mini-128k-instruct” model from Hugging Face.

from huggingface_hub import snapshot_download

model_id = "microsoft/Phi-3-mini-128k-instruct"  # Replace with the ID of the model you want to download
snapshot_download(repo_id=model_id, local_dir="phi3")

Step 2: Set Up Llama.cpp

1. Clone the Llama.cpp repository:

git clone https://github.com/ggerganov/llama.cpp

Navigate to the llama.cpp directory:

cd llama.cpp

Install the required dependencies:

pip install -r requirements.txt

This installs all the Python libraries necessary for converting models.

Step 3: Convert the Model to GGUF Format

Run the conversion script:

python llama.cpp/convert-hf-to-gguf.py ./phi3 --outfile output_file.gguf --outtype q8_0

• ./phi3: Path to the model directory.
• output_file.gguf: Name of the output file where the GGUF model will be saved.
• q8_0: Specifies the quantization type (in this case, quantized 8-bit integer).

By following these steps, you can convert a Hugging Face model to GGUF format and take advantage of the benefits of GGUF for CPU-based deployment of machine learning models. For running the model on local setups use software like ollama, lmstudio, etc gguf file are required. These files help in efficient storage of model and faster inference.