glm-4-9b-chat-IMat-GGUF

_Llama.cpp imatrix quantization of THUDM/glm-4-9b-chat_ Original Model: [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat) Original dtype: `BF16` (`bfloat16`) Quantized by: [https://github.com/ggerganov/llama.cpp/pull/6999](https://github.com/ggerganov/llama.cpp/pull/6999) IMatrix dataset: [here](https://gist.githubusercontent.com/bartowski1182/eb213dccb3571f863da82e99418f81e8/raw/b2869d80f5c16fd7082594248e80144677736635/calibration_datav3.txt) - [Files](#files) - [IMatrix](#imatrix) - [Common Quants](#common-quants) - [All Quants](#all-quants) - [Downloading using huggingface-cli](#downloading-using-huggingface-cli) - [Inference](#inference) - [Simple chat template](#simple-chat-template) - [Chat template with system prompt](#chat-template-with-system-prompt) - [Llama.cpp](#llama-cpp) - [FAQ](#faq) - [Why is the IMatrix not applied everywhere?](#why-is-the-imatrix-not-applied-everywhere) - [How do I merge a split GGUF?](#how-do-i-merge-a-split-gguf) Status: ✅ Available Link: [here](https://huggingface.co/legraphista/glm-4-9b-chat-IMat-GGUF/blob/main/imatrix.dat) If you do not have hugginface-cli installed: pip install -U "huggingface_hub[cli]" Download the specific file you want: huggingface-cli download legraphista/glm-4-9b-chat-IMat-GGUF --include "glm-4-9b-chat.Q8_0.gguf" --local-dir ./ If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run: huggingface-cli download legraphista/glm-4-9b-chat-IMat-GGUF --include "glm-4-9b-chat.Q8_0/*" --local-dir ./ [gMASK]<sop><|user|> {user_prompt}<|assistant|> {assistant_response}<|user|> {next_user_prompt} [gMASK]<sop><|system|> {system_prompt}<|user|> {user_prompt}<|assistant|> {assistant_response}<|user|> {next_user_prompt} llama.cpp/main -m glm-4-9b-chat.Q8_0.gguf --color -i -p "prompt here (according to the chat template)" According to [this investigation](https://www.reddit.com/r/LocalLLaMA/comments/1993iro/ggufs_quants_can_punch_above_their_weights_now/), it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results). 1. Make sure you have `gguf-split` available - To get hold of `gguf-split`, navigate to https://github.com/ggerganov/llama.cpp/releases - Download the appropriate zip for your system from the latest release - Unzip the archive and you should be able to find `gguf-split` 2. Locate your GGUF chunks folder (ex: `glm-4-9b-chat.Q8_0`) 3. Run `gguf-split --merge glm-4-9b-chat.Q8_0/glm-4-9b-chat.Q8_0-00001-of-XXXXX.gguf glm-4-9b-chat.Q8_0.gguf` - Make sure to point `gguf-split` to the first chunk of the split. Got a suggestion? Ping me [@legraphista](https://x.com/legraphista)!