Qwen3.6-27B-GPTQ-Pro-4bit

Welcome to **Qwen36-27B-GPTQ-Pro-4Bit** – a titan of reasoning and generation, elegantly squeezed into a remarkably efficient 4-bit package. It punches leagues above its weight class while keeping your VRAM happy and your inference speeds blazingly fast! Thank you Qwen team for another amazing model. This isn't your average quantization. We used the **GPTQ-Pro** framework combined with the **FOEM** (First-Order Error Metric) approach. This advanced technique carefully preserves the most critical weights during the 4-bit compression process by evaluating the exact impact of quantization on the model's loss landscape. The result? - **Near-Lossless Performance**: Enjoy the profound reasoning, coding prowess, and vast knowledge of a 27 Billion parameter model, but with a drastically reduced memory footprint. - **Marlin Optimized**: Ready out-of-the-box for Marlin kernels to deliver maximum token-per-second throughput in serving engines like vLLM. - **Consumer Hardware Friendly**: Fit a massive 27B powerhouse model on consumer GPUs with room to spare for massive context lengths! This repository contains a 4-bit GPTQ-Pro quantization of `unsloth/Qwen3.6-27B`, produced with GPTQModel and the FOEM/GPTAQ-style quality settings used in the `GPTQ-Pro` project. Source project: https://github.com/groxaxo/GPTQ-Pro CUDA_VISIBLE_DEVICES=0,1 vllm serve groxaxo/Qwen3.6-27B-GPTQ-Pro-4Bit \ --dtype float16 \ --quantization gptq_marlin \ --disable-custom-all-reduce \ --tensor-parallel-size 2 \ --max-model-len 132144 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --gpu-memory-utilization 0.92 CUDA_VISIBLE_DEVICES=0,1 vllm serve /path/to/Qwen3.6-27B-GPTQ-Pro-4Bit \ --dtype float16 \ --quantization gptq_marlin \ --disable-custom-all-reduce \ --tensor-parallel-size 2 \ --max-model-len 132144 from gptqmodel import BACKEND, GPTQModel model = GPTQModel.load( "groxaxo/Qwen3.6-27B-GPTQ-Pro-4Bit", backend=BACKEND.GPTQ_MARLIN, device="cuda:0", ) print(model.generate("Write a short deployment checklist.", max_new_tokens=64)[0]) - Tested with tensor parallel size 2 on RTX 3090 GPUs. - Use `float16` and `gptq_marlin` for the most reliable vLLM startup path. - The quantization and serving workflow lives in the `GPTQ-Pro` repository above. - MTP/speculative decoding is detected by vLLM for this model, but on 2x RTX 3090 the exact `--max-model-len 262144` launch OOMs during KV-cache setup. - The working local vLLM configuration I verified is `--max-model-len 65536` with `--enforce-eager`; that starts and serves, but the current metrics showed `spec_decode_num_accepted_tokens_total=0`, so it does not improve speed yet. - If you test MTP, use `--speculative-config '{"method":"mtp","num_speculative_tokens":2}'` and disable thinking in the request payload when you want a plain answer. Tested on **2× NVIDIA RTX 3090** with vLLM (gptq_marlin, tensor-parallel=2, float16). - Wikitext-2 test perplexity: **6.366** (n_ctx=1024)