gemma-4-31B-it-FP8-block

- **Model Architecture:** google/gemma-4-31B-it - **Input:** Text / Image - **Output:** Text - **Model Optimizations:** - **Weight quantization:** FP8 - **Activation quantization:** FP8 - **Release Date:** 2026-04-04 - **Version:** 1.0 - **Model Developers:** RedHatAI This model is a quantized version of [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it). It was evaluated on several tasks to assess its quality in comparison to the unquantized model. This model was obtained by quantizing the weights and activations of [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) to FP8 data type, ready for inference with vLLM. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Weights are quantized using block-wise FP8 scaling (128×128 blocks), and activations are quantized dynamically per group (group_size=128). Only the weights and activations of the linear operators within transformer blocks are quantized using [LLM Compressor](https://github.com/vllm-project/llm-compressor). Vision tower, embedding, and output head layers are kept in their original precision. This model can be deployed using [vLLM](https://docs.vllm.ai/en/latest/). For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the [Gemma 4 vLLM usage guide](https://recipes.vllm.ai/Google/gemma-4-31B-it). 1. Start the vLLM server: vllm serve RedHatAI/gemma-4-31B-it-FP8-block \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 To enable thinking/reasoning and tool calling: vllm serve RedHatAI/gemma-4-31B-it-FP8-block \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' \ --async-scheduling > **Tip:** For text-only workloads, pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip vision encoder memory allocation and free up GPU memory for a longer context window. 2. Send requests to the server: from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://<your-server-host>:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) model = "RedHatAI/gemma-4-31B-it-FP8-block" messages = [ {"role": "user", "content": "Explain quantum mechanics clearly and concisely."}, ] outputs = client.chat.completions.create( model=model, messages=messages, ) generated_text = outputs.choices[0].message.content print(generated_text) This model was created by applying data-free FP8 block quantization with [LLM Compressor](https://github.com/vllm-project/llm-compressor), as presented in the code snippet below. from llmcompressor import model_free_ptq MODEL_ID = "google/gemma-4-31B-it" SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-block" model_free_ptq( model_stub=MODEL_ID, save_directory=SAVE_DIR, scheme="FP8_BLOCK", ignore=["re:.*vision.*", "lm_head", "re:.*embed_tokens.*"], max_workers=8, device="cuda:0", ) This model was evaluated on GSM8K Platinum, MMLU-Pro, IFEval, MATH-500, AIME 2025, GPQA Diamond, and LiveCodeBench v6 using [lm-evaluation-harness](https://github.com/neuralmagic/lm-evaluation-harness) and [lighteval](https://github.com/neuralmagic/lighteval), served with [vLLM](https://docs.vllm.ai/en/latest/) (OpenAI-compatible API). All evaluations were performed with **thinking enabled**. The results were obtained using the following commands: Each benchmark was run 3 times with different random seeds (1234, 2345, 3456) and the scores were averaged; AIME 2025 used 8 seeds. **vLLM server (instruction following and reasoning benchmarks):** vllm serve RedHatAI/gemma-4-31B-it-FP8-block \ --tensor-parallel-size 2 \ --max-model-len 69632 \ --gpu-memory-utilization 0.90 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt '{"image":0,"audio":0}' \ --async-scheduling lm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=RedHatAI/gemma-4-31B-it-FP8-block,max_length=69632,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results_gsm8k_platinum.json \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234" lm_eval --model local-chat-completions \ --tasks mmlu_pro_chat \ --model_args "model=RedHatAI/gemma-4-31B-it-FP8-block,max_length=69632,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results_mmlu_pro.json \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234" lm_eval --model local-chat-completions \ --tasks ifeval \ --model_args "model=RedHatAI/gemma-4-31B-it-FP8-block,max_length=69632,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results_ifeval.json \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234" **litellm_config.yaml:** model_parameters: provider: hosted_vllm model_name: hosted_vllm/RedHatAI/gemma-4-31B-it-FP8-block base_url: http://0.0.0.0:8000/v1 api_key: '' timeout: 3600 concurrent_requests: 32 generation_parameters: temperature: 1.0 max_new_tokens: 65536 top_p: 0.95 top_k: 64 seed: 1234 Run once per seed (changing `seed` in the config each time): lighteval endpoint litellm litellm_config.yaml 'math_500|0' \ --output-dir results/ --save-details lighteval endpoint litellm litellm_config.yaml 'aime25|0' \ --output-dir results/ --save-details lighteval endpoint litellm litellm_config.yaml 'gpqa:diamond|0' \ --output-dir results/ --save-details **vLLM server:** vllm serve RedHatAI/gemma-4-31B-it-FP8-block \ --tensor-parallel-size 2 \ --max-model-len 36864 \ --gpu-memory-utilization 0.90 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt '{"image":0,"audio":0}' \ --async-scheduling **litellm_config.yaml:** model_parameters: provider: hosted_vllm model_name: hosted_vllm/RedHatAI/gemma-4-31B-it-FP8-block base_url: http://0.0.0.0:8000/v1 api_key: '' timeout: 1200 concurrent_requests: 32 generation_parameters: temperature: 1.0 max_new_tokens: 32768 top_p: 0.95 top_k: 64 seed: 1234 Run once per seed: lighteval endpoint litellm litellm_config.yaml 'lcb:codegeneration_v6|0' \ --output-dir results/ --save-details