Kimi-K3-DSpark

A long-context DSpark speculator for [Kimi K3](https://huggingface.co/moonshotai/Kimi-K3). It supports context lengths of up to 1 million tokens. A DSpark speculator for the Kimi K3 target, enabling faster inference through speculative decoding. DSpark extends the DFlash parallel-draft backbone with a Markov logit-bias head and a per-position confidence head. This checkpoint was trained with [SpecForge](https://github.com/sgl-project/SpecForge/) using hidden states from a live SGLang target engine. - **Base model:** `moonshotai/Kimi-K3` - **Format:** Safetensors (single-file BF16, 2,249,289,601 parameters) - **Draft:** 5 full-attention Qwen3-style GQA layers, hidden size 7168, 64 query heads / 16 KV heads, and `block_size=7` - **Verification width:** 1 current token + 7 draft tokens - **Auxiliary target layers:** `[7, 23, 51, 67, 83]` - **Trained context:** 65,536 tokens - **Target weights:** embedding and unembedding weights are not included `acc_len` is SGLang's histogram-native request acceptance length, averaged within each question and then equally across questions. RULER V2 uses the 1M input configuration. Actual prompts span 1,000,432–1,047,925 tokens; partition acc_len is 4.4658 for MK, 4.3081 for MV, and 3.9919 for QA. [SGLang Cookbook](https://lmsysorg.mintlify.app/cookbook/autoregressive/Moonshotai/Kimi-K3#hw=b300&pdMode=unified&strategy=low-latency&spec=dspark&hicache=off) provides Kimi K3 deployment recipes. sglang serve \ --trust-remote-code \ --model-path moonshotai/Kimi-K3 \ --tp-size 8 \ --dcp-size 8 \ --mem-fraction-static 0.85 \ --max-mamba-cache-size 160 \ --max-running-requests 32 \ --cuda-graph-max-bs-decode 32 \ --reasoning-parser kimi_k3 \ --tool-call-parser kimi_k3 \ --host 0.0.0.0 \ --port 30000 \ --speculative-algorithm DSPARK \ --speculative-draft-model-path RadixArk/Kimi-K3-DSpark \ --speculative-dspark-block-size 7 \ --speculative-draft-attention-backend trtllm_mha \ --enable-linear-replayssm-spec \ --context-length 1048576 \ --chunked-prefill-size 16384 YaRN-16 is enabled in the published draft config by default with `original_max_position_embeddings=65536` and `max_position_embeddings=1048576`; no separate draft config override is required. - **Framework:** SpecForge online distillation, with hidden states captured from a frozen Kimi K3 target served by a live SGLang engine. Draft trained from random initialization. - **Loss:** `0.1 CE + 0.9 L1 distillation + 1.0 confidence BCE`, decay gamma 4.0, with 512 sampled anchors per sequence and `block_size=7`. - **Topology:** 4 nodes × 4 GB300 (16 ranks) — 2 × TP8 target replicas, DP2 sampler, FSDP16 `SHARD_GRAD_OP` on the draft, TP-batch scatter. Batch 8 per replica × 32 accumulation steps × 2 replicas = global batch 512.