deepseek-v4-gguf

This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader). https://github.com/antirez/ds4 Use **q2** on 128 GB Mac machines, **q4** on machines with ≥ 256 GB RAM, pair either with **MTP** for optional speculative decoding. The filename is the spec. In detail, for the **q2** file: For the **q4** file, only the three routed-expert classes change to `Q4_K`. Everything else is byte-for-byte identical to the q2 recipe. The motivation behind the asymmetry: the routed experts are the majority of the parameter count but each individual expert handles only a fraction of tokens, so aggressive quantization on them costs less in average quality than the same treatment of router, projections, or shared experts. Keeping the decision-making components at `Q8_0` preserves model behavior; crushing the experts buys the size. git clone https://github.com/antirez/ds4 cd ds4 ./download_model.sh q2 # 128 GB RAM machines ./download_model.sh q4 # >= 256 GB RAM machines ./download_model.sh mtp # optional MTP / speculative decoding make ./ds4 -p "Explain Redis streams in one paragraph." ./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192 The `download_model.sh` script fetches from this repository, resumes partial downloads, and points `./ds4flash.gguf` at the selected variant. MIT. The base model copyright is held by DeepSeek; the GGUFs are redistributed under the base model's release terms.