Which models and hardware does DeepInfer support?

DeepInfer's first open release supports one model, DeepSeek V4.1 Flash, and downloads the DeepSeek-V4.1-Flash-1017 checkpoint from Hugging Face by default. It runs on NVIDIA SM100 GPUs with CUDA 13 and on Huawei Ascend 950 NPUs with CANN 9.2, on Linux with GCC 13 or newer and Python 3.12.

Can DeepInfer replace a full production serving stack today?

Not yet. The DeepInfer README says this release provides offline inference with the core features for serving at scale, but leaves out a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. Those parts are listed as required for a complete inference service and are not included in this release.

How does DeepInfer relate to vLLM and SGLang?

DeepSeek says DeepInfer is built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM. It relies on DeepSeek's own kernel libraries, including FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom TileLang kernels. The Q&A notes the public FlashMLA decode kernel is 5–10% slower than the internal one, and end-to-end speed differs by at most 5%.

How much does DeepSeek's DSpark speculative decoding help?

According to DeepInfer's deployment Q&A, DSpark's confidence-scheduled verification cuts verified tokens by about 30% compared with a fixed five-token draft while keeping accepted length almost the same. DeepSeek reports +4–7% throughput at about 60 concurrent sequences per rank, +11.0% at 96, and a small −1.5% loss at 16.

How many GPUs does DeepSeek recommend for an efficient deployment?

DeepInfer's Q&A says a one-prefill, one-decode setup is possible, but the efficient minimum is 64 prefill GPUs plus 64 decode GPUs, because DeepSeek recommends running decode at expert parallelism of EP64 or above. The published benchmark charts use 4 SM100 GPUs for prefill and 64 for decode.

DeepSeek · 2026-10-20 · major

DeepInfer — DeepSeek open-sources the inference engine behind its API

DeepInfer is DeepSeek's Apache-2.0 inference framework, the engine that powers its production services. The first open release runs DeepSeek V4.1 Flash offline on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

GitHub card for the deepseek-ai/DeepInfer repository

DeepSeek publishes the high-performance engine that serves its own models, starting with offline inference for V4.1 Flash.

Key specs

GitHub stars	7.5k

Quick facts

Maker	DeepSeek
License	Apache-2.0
Supported model	DeepSeek V4.1 Flash (default checkpoint DeepSeek-V4.1-Flash-1017)
Hardware	NVIDIA SM100 GPUs, Huawei Ascend 950 NPUs
Requirements	Linux, GCC 13+, Python 3.12, CUDA 13 or CANN 9.2
Not included yet	KV cache system, prefill/decode disaggregation, DP load balancing

What is it?

DeepInfer is the inference framework that runs DeepSeek's large-scale production services, now released as open source under Apache-2.0. The first release covers offline inference for DeepSeek V4.1 Flash, with optimized kernels, a model and sampler runtime, and the parallelism DeepSeek uses at scale. A long Q&A document in the repo also explains how DeepSeek deploys the full service.

How does it work?

The engine combines attention data parallelism, balanced context parallelism and MoE expert parallelism with expert load balancing (EPLB). Its DSpark speculative decoding uses a confidence head to decide how many draft tokens to verify, SWA bounded replay recomputes only the last 128 tokens on a cache hit, and an asynchronous batch scheduler keeps the GPU busy while the CPU plans the next step. It also offers batch-invariant and deterministic sampling on NVIDIA, plus structured outputs from regex, JSON schema or EBNF grammars.

Why does it matter?

Teams that serve DeepSeek open weights can now run the same engine DeepSeek uses, instead of reproducing its tricks in vLLM or SGLang. The Q&A shares production numbers rarely made public, such as decode at EP64 or larger and about 100 PiB of KV cache storage. The project is built mainly on vLLM, so improvements may also flow back to the wider serving ecosystem.

Who is it for?

ML infrastructure engineers serving DeepSeek models at scale

Frequently asked questions

Which models and hardware does DeepInfer support?: DeepInfer's first open release supports one model, DeepSeek V4.1 Flash, and downloads the DeepSeek-V4.1-Flash-1017 checkpoint from Hugging Face by default. It runs on NVIDIA SM100 GPUs with CUDA 13 and on Huawei Ascend 950 NPUs with CANN 9.2, on Linux with GCC 13 or newer and Python 3.12.
Can DeepInfer replace a full production serving stack today?: Not yet. The DeepInfer README says this release provides offline inference with the core features for serving at scale, but leaves out a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. Those parts are listed as required for a complete inference service and are not included in this release.
How does DeepInfer relate to vLLM and SGLang?: DeepSeek says DeepInfer is built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM. It relies on DeepSeek's own kernel libraries, including FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom TileLang kernels. The Q&A notes the public FlashMLA decode kernel is 5–10% slower than the internal one, and end-to-end speed differs by at most 5%.
How much does DeepSeek's DSpark speculative decoding help?: According to DeepInfer's deployment Q&A, DSpark's confidence-scheduled verification cuts verified tokens by about 30% compared with a fixed five-token draft while keeping accepted length almost the same. DeepSeek reports +4–7% throughput at about 60 concurrent sequences per rank, +11.0% at 96, and a small −1.5% loss at 16.
How many GPUs does DeepSeek recommend for an efficient deployment?: DeepInfer's Q&A says a one-prefill, one-decode setup is possible, but the efficient minimum is 64 prefill GPUs plus 64 decode GPUs, because DeepSeek recommends running decode at expert parallelism of EP64 or above. The published benchmark charts use 4 SM100 GPUs for prefill and 64 for decode.

Try it

export DEEPINFER_ACCELERATOR=nvidia && ./build.sh && python examples/offline_inference.py