GadflyII/GLM-4.7-Flash-NVFP4

Zhipu

https://github.com/Gadflyii/vllm/tree/main This is a **mixed precision NVFP4 quantization** of [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash), a 30B-A3B (30B total, 3B active) Mixture-of-Experts model. This model was made via custom quantization and calibration (128 samples, 2048 max seq len, neuralmagic/calibration, all 64 experts) scripts based on NVIDIA's approach for DeepSeek-V3. It uses **mixed precision** to preserve accuracy: - **vLLM**: 0.14.0+ (for compressed-tensors NVFP4 support) - **transformers**: 5.0.0+ (for `glm4_moe_lite` architecture) - **GPU**: NVIDIA GPU with FP4 tensor core support (Blackwell, Hopper, Ada Lovelace) pip install vllm>=0.14.0 pip install git+https://github.com/huggingface/transformers.git from vllm import LLM, SamplingParams model = LLM( "GadflyII/GLM-4.7-Flash-NVFP4", tensor_parallel_size=1, max_model_len=4096, trust_remote_code=True, gpu_memory_utilization=0.85, ) params = SamplingParams(temperature=0.7, max_tokens=512) outputs = model.generate(["Explain quantum computing in simple terms."], params) print(outputs[0].outputs[0].text) vllm serve GadflyII/GLM-4.7-Flash-NVFP4 \ --tensor-parallel-size 1 \ --max-model-len 4096 \ --trust-remote-code - **Base Model**: [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) - **Architecture**: `Glm4MoeLiteForCausalLM` - **Parameters**: 30B total, 3B active per token (30B-A3B) - **MoE Configuration**: 64 routed experts, 4 active, 1 shared expert - **Layers**: 47 - **Context Length**: 202,752 tokens (max) - **Languages**: English, Chinese - **Format**: compressed-tensors (NVFP4) - **Block Size**: 16 - **Scale Format**: FP8 (E4M3) - **Calibration**: 128 samples from neuralmagic/calibration dataset - **Full Expert Calibration**: All 64 experts calibrated per sample If you use this model, please cite the original GLM-4.7-Flash: @misc{glm4flash2025, title={GLM-4.7-Flash}, author={Zhipu AI}, year={2025}, howpublished={\url{https://huggingface.co/zai-org/GLM-4.7-Flash}} } This model inherits the Apache 2.0 license from the base model.