DeepSeek-V4-Flash-DSpark

**Note: DeepSeek-V4-Flash-DSpark** is **not** a new model. It is the same checkpoint with an additional speculative decoding module attached. A minimal inference example is available in the [`inference`](inference/README.md) folder. For more details, refer to: https://github.com/deepseek-ai/DeepSpec We present a preview version of **DeepSeek-V4** series, including two strong Mixture-of-Experts (MoE) language models — **DeepSeek-V4-Pro** with 1.6T parameters (49B activated) and **DeepSeek-V4-Flash** with 284B parameters (13B activated) — both supporting a context length of **one million tokens**. DeepSeek-V4 series incorporate several key upgrades in architecture and optimization: 1. **Hybrid Attention Architecture:** We design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency. In the 1M-token context setting, DeepSeek-V4-Pro requires only **27% of single-token inference FLOPs** and **10% of KV cache** compared with DeepSeek-V3.2. 2. **Manifold-Constrained Hyper-Connections (mHC):** We incorporate mHC to strengthen conventional residual connections, enhancing stability of signal propagation across layers while preserving model expressivity. 3. **Muon Optimizer:** We employ the Muon optimizer for faster convergence and greater training stability. We pre-train both models on more than **32T** diverse and high-quality tokens, followed by a comprehensive post-training pipeline. The post-training features a two-stage paradigm: independent cultivation of domain-specific experts (through SFT and RL with GRPO), followed by unified model consolidation via on-policy distillation, integrating distinct proficiencies across diverse domains into a single model. **DeepSeek-V4-Pro-Max**, the maximum reasoning effort mode of DeepSeek-V4-Pro, significantly advances the knowledge capabilities of open-source models, firmly establishing itself as the best open-source model available today. It achieves top-tier performance in coding benchmarks and significantly bridges the gap with leading closed-source models on reasoning and agentic tasks. Meanwhile, **DeepSeek-V4-Flash-Max** achieves comparable reasoning performance to the Pro version when given a larger thinking budget, though its smaller parameter scale naturally places it slightly behind on pure knowledge tasks and the most complex agentic workflows. *\*FP4 + FP8 Mixed: MoE expert parameters use FP4 precision; most other parameters use FP8.* DeepSeek-V4-Pro and DeepSeek-V4-Flash both support three reasoning effort modes: This release does not include a Jinja-format chat template. Instead, we provide a dedicated `encoding` folder with Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model, and how to parse the model's text output. Please refer to the [`encoding`](encoding/README.md) folder for full documentation. A brief example: from encoding_dsv4 import encode_messages, parse_message_from_completion_text messages = [ {"role": "user", "content": "hello"}, {"role": "assistant", "content": "Hello! I am DeepSeek.", "reasoning_content": "thinking..."}, {"role": "user", "content": "1+1=?"} ] prompt = encode_messages(messages, thinking_mode="thinking") import transformers tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Pro") tokens = tokenizer.encode(prompt) DSpark speculative decoding is enabled with a single flag — add --speculative-config with method: dspark to your vLLM launch command: `--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'` For example, the command below serves the model with vLLM on a single 4×GB300 node. See the [vLLM recipe](https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash?hardware=b300&features=tool_calling,reasoning) for detailed instructions and other hardware configurations. vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \ --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \ --data-parallel-size 4 --enable-expert-parallel \ --moe-backend deep_gemm_mega_moe \ --attention-config '{"use_fp4_indexer_cache": true}' \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}' Please refer to the [inference](inference/README.md) folder for detailed instructions on running DeepSeek-V4 locally, including model weight conversion and interactive chat demos. For local deployment, we recommend setting the sampling parameters to `temperature = 1.0, top_p = 1.0`. For the Think Max reasoning mode, we recommend setting the context window to at least **384K** tokens. This repository and the model weights are licensed under the [MIT License](LICENSE). @misc{deepseekai2026deepseekv4, title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence}, author={DeepSeek-AI}, year={2026}, } If you have any questions, please raise an issue or contact us at [service@deepseek.com](service@deepseek.com).