endless-frontier_BigBang-v1-GGUF
Using <a href="https://github.com/ggml-org/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggml-org/llama.cpp/releases/tag/b10262">b10262</a> for quantization. Original model: https://huggingface.co/endless-frontier/BigBang-v1 **Model details:** - Parameter count: 36B - Input support: text, image (with mmproj file) - [details](#multimodal) - MTP: yes - [details](#mtp) - imatrix: yes - [details](#imatrix) [How to run](#how-to-run) {system_prompt}<|im_end|> {prompt}<|im_end|> **Don't know which to choose?** Grab [Q4_K_M](https://huggingface.co/bartowski/endless-frontier_BigBang-v1-GGUF/blob/main/endless-frontier_BigBang-v1-Q4_K_M.gguf) (21.86GB) - usually a good mix of size and performance. Download instructions available [here](#downloading-using-the-hugging-face-cli) Download a specific file: hf download bartowski/endless-frontier_BigBang-v1-GGUF --include "endless-frontier_BigBang-v1-Q4_K_M.gguf" --local-dir ./ First, make sure you have the Hugging Face CLI installed: pip install -U "huggingface_hub[cli]" Download a specific file: hf download bartowski/endless-frontier_BigBang-v1-GGUF --include "endless-frontier_BigBang-v1-Q4_K_M.gguf" --local-dir ./ The files marked `true` in the Split column above are stored as multiple parts in a folder. To download all the parts to a local folder, run: hf download bartowski/endless-frontier_BigBang-v1-GGUF --include "endless-frontier_BigBang-v1-bf16/*" --local-dir ./ You can either specify a new local-dir (endless-frontier_BigBang-v1-bf16) or download them all in place (./) These quants run with [llama.cpp](https://github.com/ggml-org/llama.cpp) - installable in one line via [llama.app](https://llama.app/): curl -LsSf https://llama.app/install.sh | sh llama-server -hf bartowski/endless-frontier_BigBang-v1-GGUF:Q4_K_M llama-server includes a built-in chat web UI, served at http://localhost:8080 by default. These quants were made with llama.cpp release b10262 - if this model's architecture is newly supported, you'll need that release or newer to run them. They also work in: [LM Studio](https://lmstudio.ai/) · [koboldcpp](https://github.com/LostRuins/koboldcpp) · [ramalama](https://github.com/containers/ramalama) · [Jan AI](https://www.jan.ai/) · [Text Generation Web UI](https://github.com/oobabooga/text-generation-webui) · [LoLLMs](https://github.com/ParisNeo/lollms) · [Atomic Chat](https://atomic.chat/) This model supports image input. Alongside the quants, this repo includes the multimodal projector files [mmproj-endless-frontier_BigBang-v1-f16.gguf](https://huggingface.co/bartowski/endless-frontier_BigBang-v1-GGUF/blob/main/mmproj-endless-frontier_BigBang-v1-f16.gguf) and [mmproj-endless-frontier_BigBang-v1-bf16.gguf](https://huggingface.co/bartowski/endless-frontier_BigBang-v1-GGUF/blob/main/mmproj-endless-frontier_BigBang-v1-bf16.gguf), which pair with any quant above. llama.cpp downloads the mmproj automatically when using `-hf` as shown above; if you're loading files manually, pass it with `--mmproj`. This model has MTP (Multi-Token Prediction) layers, and they are included in these quants MTP layers act as a built-in draft model, letting llama.cpp run speculative decoding for faster generation. To use them, add the following flag to your llama.cpp command: --spec-type draft-mtp Note: the MTP layers are stored at Q4_0 in the imatrix quants (except for the Q8_0 quant), since imatrix calibration does not exercise them. Q4_0 is chosen for its speed which massively benefits MTP performance. All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/82ae9b520227f57d79ba04add13d0d0d). The imatrix is available here: [endless-frontier_BigBang-v1-imatrix.gguf](https://huggingface.co/bartowski/endless-frontier_BigBang-v1-GGUF/blob/main/endless-frontier_BigBang-v1-imatrix.gguf). Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to. llama.cpp automatically "repacks" weights into an interleaved layout at load time for faster inference on ARM and AVX machines - details in [this PR](https://github.com/ggml-org/llama.cpp/pull/9921). This once required downloading special Q4_0_4_4/4_8/8_8 files; those are long gone. Online repacking now covers Q4_0, IQ4_NL, and most K-quants, so no special quant choice is needed for CPU inference. An older (early 2024) but still useful write-up with charts comparing quant performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9) The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have. If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM. If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total. Hugging Face can also do this math for you: add your hardware in your [Local Apps settings](https://huggingface.co/settings/local-apps) and the model page will show which files fit. Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'. If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M. If you want to get more into the weeds, you can check out this extremely useful feature chart: [llama.cpp feature matrix](https://github.com/ggml-org/llama.cpp/wiki/Feature-matrix) But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size. These I-quants can also be used on CPU, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide. Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset. Thank you ZeroWw for the inspiration to experiment with embed/output. Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski
Open endless-frontier_BigBang-v1-GGUF in AIniverse
Compare versions, read real ratings, save to your stack.
Open in App