DeepSeek Open-Sources Inference Framework DeepInfer

DeepSeek announced on October 20 that it had open-sourced DeepInfer, an inference framework it says powers its DeepSeekV4.1Flash production services, and released a companion tool for token encoding and decoding. The company said the release is intended to help developers run its models more efficiently and reduce computing costs and energy use. DeepInfer is available under the Apache 2.0 license. Its project documentation describes support for optimized kernels, distributed inference, speculative decoding, asynchronous scheduling, deterministic sampling and structured outputs. It lists NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs as supported hardware. The documentation also specifies that the release does not include a key-value cache system or prefill-and-decode disaggregation, functions required for a complete inference service. DeepSeek describes the framework as designed for offline inference and lists DeepSeekV4.1Flash as its supported model. It says the software was built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. The companion tool, deepseekrecipe, handles token encoding and decoding. DeepSeek said it is aligned with the processing...