DeepSeek V4.1 Flash: Large-Scale Inference Deployment Practice

The capability boundaries of a large model are defined by the model itself, and the core problem that inference frameworks need to solve is how to enable these capabilities to serve a massive number of users at a reasonable cost.

Faced with massive AI invocation demands, inference frameworks have evolved from standalone service programs into complex systems that collaborate across clusters. Operators, communication, scheduling, and cache management interact and jointly determine end-to-end performance. For large-scale online services, performance optimization needs to be tailored to specific deployment conditions: what chips to use, how to organize the hardware, how to allocate computation, and how to reuse existing computation results.

From its inception, the DeepSeek V4.1 Flash inference system established three goals:

1. Compatible with computing chips from multiple manufacturers: Supports deployment on different hardware platforms and fully leverages the performance of each platform.
2. Based on common hardware organization, Prefill and Decode: Use common 8-card machines for Prefill and common 64-128-card supernodes for Decode, and design computation, communication and scheduling around the load characteristics of the two stages.
3. Supports retaining KVCache for more than 72 hours: provides a longer time window for context reuse, reduces redundant prefill calculations when the cache is hit, and incorporates the capacity and management overhead required to retain the cache for a long time into the system design.

To achieve these goals, we continuously optimize computing, communication, scheduling, and storage to improve the system's effective throughput while meeting latency requirements, supporting more requests with less computing power and energy, and enabling intelligent services to reach more users at a lower price.

Previously, we open-sourced TileLang for developing high-performance operators and a series of inference operator libraries. Today, we officially open-source DeepInfer, a large-scale cloud service inference framework . Based on this framework, we have implemented a high-throughput, low-latency large-scale inference service for DeepSeek V4.1 Flash. Currently, the service throughput of a single cluster is 155 trillion tokens/day, and the overall revenue is $38 per card per hour (based on off-peak prices).

DeepInfer integrates high-performance operators and a deeply optimized inference engine, supporting capabilities such as distributed inference, speculative decoding, SWA Bounded Replay, and structured output. It can run on NVIDIA and Huawei Ascend platforms. Simultaneously, we have open-sourced the production-grade token encoding/decoding component deepseek-recipe, supporting efficient and accurate encoding/decoding between mainstream large model API protocols and model tokens. Its processing logic is fully aligned with the official DeepSeek API, ensuring that third-party deployments receive the same API experience as official deployments. In the future, the inference code for new DeepSeek models and subsequent performance optimizations will also be continuously open-sourced within DeepInfer.

The KVCache management and load balancing components in the inference system are deeply coupled with the internal infrastructure, and their code is not yet fully open source. This article will start from the above design goals, introduce the overall architecture and key implementation of the DeepSeek V4.1 Flash inference system, as well as the design details of these two parts.

Overview of Reasoning System Design

The optimization goals of the DeepSeek V4.1 Flash inference system are: higher throughput and lower latency. To achieve these two goals, we use:

• (Super)node Intranode MoE Partial Expert Parallelism (EP): Increase batch size, improve GPU matrix multiplication efficiency, and thus increase throughput.
• Vision Encoder / Prefill / Decode Separate Deployment: Decouple resource allocation for each stage, independently control and optimize TTFT and TBT respectively.

Inheriting the DeepSeek-V3/R1 inference system, the ATTN part uses data parallel DP (#DP=#EP). Load balancing is required between different DPs/EPs. For this purpose, we developed Prefill Load Balancer, Decode Load Balancer, Vision Encoder Load Balancer, and Expert- Parallel Load Balancer. To save the cost of repeated prefilling, we developed KVCache Storage.

Before and after a user requests inference, a conversion between the API protocol and the token sequence is required. DeepSeek Recipe supports efficient and accurate encoding and decoding between mainstream large model API protocols and model token sequences. Through special token segmentation and parallel strategies, it avoids special token injection while reducing the processing latency of long text tokenization.

Please refer to the following architecture diagram:

Parallel strategy:

• Prefill: EP2～8/DP2～8; Use the smallest possible EP/DP ratio when video memory is sufficient.
• Decode: EP64~128/DP64~128 (Wide-EP); Use the largest possible EP/DP if the graphics card is available.
• Vision Encoder: Generally, a single-card deployment is sufficient. It is recommended to use the cheapest available card for deployment.

Actual statistical data of the DeepSeek-V4.1-Flash online system

Offline performance benchmark data

DeepSeek Recipe

API multi-protocol adaptation

DeepSeek Recipe converts requests from the Chat Completions API, Responses API, and Messages API into a single dialogue format, sharing model templates and encoding logic, and generating prompt and token sequences based on the model version.

After parsing, the model output is divided into text, thought content, and tool calls, which are then converted into streaming responses for various protocols. Request transformation, prompt encoding, and output parsing are all implemented in Rust for efficient processing, while also supporting Python binding calls to improve ease of use.

Tokenizer processing and acceleration

In practice, Tokenizer faces two challenging issues: (1) As the model context lengthens, the linear tokenization method introduces significant latency, impacting the Time-To-Function (TTFT) and significantly increasing the CPU overhead of the service; (2) Special tokens are used to mark message roles, thought boundaries, and tool calls, but these tokens may also appear as ordinary text in the dialogue. When processing requests, it is necessary to avoid the special token text carried in user input from negatively impacting the model's inference performance. To address these two issues, we implemented a solution based on special token boundary matching in DeepSeek Recipe.

Input Encoding: When constructing the prompt, explicitly inserted special tokens and their positions are recorded. During encoding, these are directly mapped to their corresponding token IDs, while ordinary text is encoded using the basic vocabulary. Strings such as `<input>` and `<output>` in user input, tool returns, and tool call parameters <｜User｜>are <｜image｜>encoded into corresponding ordinary token sequences to prevent text injection from controlling tokens and influencing model behavior. Output Parsing: When processing the response token stream, we identify special tokens </think>in the "End of Thought" marker and DSML tags based on their token IDs ｜DSML｜. The same text concatenated by the model using ordinary tokens is correctly preserved as the original content and does not gain the ability to trigger the end of thought or tool call using special tokens. Segmentation Acceleration: The positions and boundaries of special tokens are already determined when constructing the prompt. Based on the encoding principle of BPE tokenizer, it can be found that the segments of ordinary text between special tokens can be encoded independently. Therefore, during encoding, we first skip the matching and segmentation of special tokens in the ordinary text, then encode each segment of ordinary text in parallel, and finally concatenate the results in the original order. This reduces the word segmentation time of ultra-long context requests and optimizes request latency.

Streaming output analysis

The inference backend returns chunks at the token level. A single UTF -8 code point, DSML tag, or tool call parameter may be split into multiple chunks. To correctly decode the model's output and parse its structure as early as possible to reduce return latency, we need to implement a cross-chunk streaming parsing state machine. This machine should retain the processing progress and state of incomplete tasks in memory and return the determined content promptly.

Cross-chunk parsing: When encountering incomplete UTF-8 code points during decoding, the state machine buffers the relevant tokens. Once subsequent tokens arrive and the state is complete, the state is handed over to the parser. The parser maintains its state according to stages such as thought, main text, and tool parameters, incrementally matching delimiters using the KMP algorithm. If only part of the tag is found at the end of a chunk, the matching progress is temporarily preserved, and incremental recognition continues when the next chunk arrives. Confirmed content is output along with the parsing progress, eliminating the need to repeatedly scan previous text. Tool parameter conversion: Once the tool name is parsed, it returns immediately, and subsequent JSON fragments are directly generated based on DSML parameter boundaries. String values ​​are JSON-escaped segment by segment, while non-string values ​​such as arrays and objects are output as raw JSON text, eliminating the need to construct complete parameter objects and saving the overhead of repeated parsing and serialization. Even with long parameters, they can be returned segment by segment during generation, shortening the waiting time for the first parameter fragment.

Load balancing

Both data parallelism and expert parallelism require distributing a balanced load across each card as much as possible to reduce the overhead of idle time and waiting caused by card asynchrony. Specifically, the Prefill/Decode Load Balancer controls the load balancing of the attention through request routing and scheduling; the Vision Encoder Load Balancer controls the load balancing between different vision encoders; and the Expert-Parallel Load Balancer controls the load balancing of the MoE sublayer through redundant experts and dynamic adjustments.

Prefill Load Balancer

The implementation mechanism of Prefill Load Balancer can be divided into 4 layers. The idea is to route and distribute requests based on the computational cost of the fitted attention sublayer (denoted as compute-quota, which is the pre-fitted quadratic cost model cost(q, c), q= the number of unhitted tokens in the sequence, c= the length of the prefix that has been computed in this sequence).

Bucketing: Requests are divided into four tiers based on the total input length: <16K / 16-64K / 64-256K / ≥256K. The number of engines allocated to each tier is dynamically redistributed based on the proportion of compute-quota demand in the previous round for that tier. Inter-DP Groups: Requests are pulled based on the remaining compute-quota. Since any engine can hit any prefix, prefix affinity routing is not performed. Intra-DP Groups: After a batch of requests is sent to an engine, each received request is distributed to the DP rank with the lowest current compute-quota load. Intra-card: Requests are dequeued FIFO. The chunk size is not fixed and is determined by inverse calculation of the budget. Each step has a time budget (in nanoseconds). During scheduling, a fitted quadratic cost model is used to predict how long each step will take. The maximum time budget that perfectly satisfies the remaining time budget is then calculated cost(q, c)by qinverse calculation of this quadratic equationq.

Decode Load Balancer

The Decode Load Balancer performs unified routing for all cards in all decode instances. The principle is to balance the number of requests and the total length of the KVCache at the same time. The former determines the batch size at each step, and the latter determines the cost of the attention part. The Decode Load Balancer heuristically sends short requests to cards with lower concurrency and larger total KVCache length, and long requests to cards with smaller total KVCache length. For details, please refer to the DualPath intra-group scheduling algorithm [1].

Vision Encoder Load Balancer

Image routing is performed on a per-machine basis. The prefill machine randomly selects a machine with a number of vision encoders deployed on it. On this machine, the image is assigned to (width/patch_size) × (height/patch_size)the card with the fewest current image patches (calculated directly from the width and height) to generate the embedding.

Expert-Parallel Load Balancer

The implementation method of expert parallel load balancing can be summarized as follows: each card has several redundant experts in addition to the resident experts to house copies of popular experts. By statistically analyzing the load of each expert over the most recent N steps, experts are dynamically adjusted and redistributed to cards every M steps based on the statistical results. The expert weights are transferred within the parallel group in the scale-up network via symm_mem.

Prefill strategy: Start with 4 redundant experts per card (padded up to a multiple of ep_size in total number of experts); statistical window 100 steps, re-ranked every 1000 steps; online data measured moe_lb (slowest rank / mean) = 1.0320 (EP2/DP2) / 1.0618 (EP4/DP4). Decode strategy: Start with 1 redundant expert per card (padded up to a multiple of ep_size in total number of experts), for example, after rounding EP64, there are 7 experts per card (excluding shared experts ); statistical window 1000 steps, re-ranked every 10000 steps; online data measured moe_lb = 1.090.

Automatic expansion and contraction

After PD is separated, the resource demand ratio between the two changes dynamically with traffic. The resource mismatch under static allocation leads to queuing and idle time. Therefore, it is necessary to adjust the ratio of Prefill/Decode instances at the cluster level according to traffic through automatic scaling up and down.

Implementation: Prefill determines the required number of instances by counting the number of missing tokens, while decode determines the required number of instances by counting the number of requests being processed, adjusting the rate limit and upper and lower bounds accordingly. To address the resource waste caused by long-tail requests during large EP deployments, decode instances enter a grace period (1 minute) after receiving a shutdown notification. Requests and their KVCaches that are still incomplete after the grace period are forwarded to other instances to continue decoding. Redundant Quota: Redundant quotas are allocated during deployment to handle traffic fluctuations. This includes two aspects: first, unused ready machines are kept in the pool as expansion inventory; second, there are always more instances online than the calculated demand. Specifically, scaling down is prohibited when the quota occupancy rate is higher than 0.8. Fault Tolerance Mechanism: Error restarts occur on the order of minutes. Fault tolerance relies on a combination of quota redundancy and request-level rerouting, rather than shortening instance restart time. In large EP deployments, all cards of an instance constitute a single communication domain; failure of any one card renders the entire instance unusable. During a restart, in order to continue using the remaining healthy cards of the instance, the platform redetermines the EP size based on all currently healthy cards, and the instance uses the actual number of cards it receives as the EP size.

KVCache Storage

With the growth of dialogue context and the widespread use of agents, KVCache prefix cache reuse can greatly reduce prefill recalculation, thereby significantly reducing costs. The DeepSeek-V4.1-Flash model's KVCache storage is divided into two parts: Block KVCache and State KVCache, stored in the 3FS cluster and local memory respectively. Block KVCache: Managed in paginated blocks of 128 tokens, storing the main KV and indexer K across four layers, according to the model architecture. SWA KVCache: Each of the first 20 layers has a 134-token ring (window length = 128 + speculative backoff = 6), and does not store compressed intermediate states.

Block KVCache

Block KVCache persists to disk on a dedicated 3FS storage cluster, without using local memory storage. Compared to multi-level storage systems, the dedicated storage cluster ensures that all prefill machines can hit any prefix, avoiding conflicts between prefix affinity and load balancing routing strategies. It also decouples storage and compute nodes from automatic scaling, and KVCache is unaffected by compute node failures (compute node failure rates are orders of magnitude higher than storage failure rates). The KVCache system design considers two factors: increasing storage capacity to improve hit rate and increasing bandwidth to reduce read latency.

In terms of capacity, the 3FS distributed file system [2] provides approximately 100 PiB of SSD storage for KVCache. The TTL (Time-To-Live) of agent traffic KVCache reaches 3 days, and that of chatbot traffic reaches 7 days. For chatbot, the long-tail hit is more severe, and a small number of KVCaches will be read many times. After missing, the overall hit rate will be greatly reduced. According to experience, a TTL of about 3 days will cause the KVCache hit rate of chatbot traffic to decrease by 0.5%-1%. In terms of bandwidth, the 3FS storage cluster is optimized for random read and write at the KVCache block granularity (approximately 112 KiB). The peak aggregate read bandwidth reaches 10 TiB/s, and the peak aggregate write bandwidth is approximately 850 GiB/s.

At the inference framework level, Block KVCache is stored on disk by decode instances, allowing both prefill and decode outputs to be reused. Regarding KVCache hit overhead, the hit path is divided into three segments, with only the first segment on the critical path: matching is performed by the Entrypoint on the kvcache-store of the local CHECK role, using hash chain prefix matching. Only metadata is retrieved, but the hit length needs to be calculated before routing and cannot be masked; KV retrieval is done asynchronously in a separate process, moving the KVCache from 3FS to the host machine shm; the transfer from shm to HBM is done in a thread within the engine process, aiming to achieve overlapping of layer-by-layer H2D (Host-to-Device) transmission and forward computation. Due to the poor performance of the CUDA copy engine for fine-grained copying of KVCache Blocks, GDR is used for the transfer, allowing full utilization of PCIe bandwidth with an RDMA network card without consuming SM.

SWA KVCache

SWA KV is not stored in the persistent cache, but remains in the memory of the machine that generated it. A memory directory that is logically unified but actually stored separately by key on each Entrypoint records which machine it is on. If the local machine does not have the key when it is hit, it is retrieved from another machine. Since SWA KV is only reused within a minute-level window in a session, the total capacity of the pool is small but the turnover is fast, which is enough to cover simultaneously active sessions. Online data shows that its TTL is about 180 seconds and the hit rate is over 98%. Misses caused by eviction are recalculated by the above-mentioned Encoder SWA Bounded Replay [3].

SWA KVCache only stores at positions that are multiples of 128 tokens. Storage occurs after the Prefill phase: the position is taken from the end of the sequence, leaving one token, and then aligned down to the position equal to 128. If the position is less than one 128 token, it is not stored in this round. Chunked prefill does not store during the process; only the last chunk is stored. During the Decode phase, a separate copy area is created in GPU memory. Each time a 128-token boundary is crossed, the current window of the request is copied from GPU memory into the copy area. When the request ends, it is stored at the position of the most recently copied element in the copy area, with the attachment point aligned down to the 128-token boundary. The copy position can cross a maximum of 5 tokens beyond the boundary, absorbed by the 6 extra positions in the ring compared to the window.

Hit Strategy

The hit principle is as follows: the hit length is determined by the Block KVCache, and whether the last block needs to be recalculated is determined by the SWA KVCache. If the SWA KVCache misses, a block is backed up and recalculated. Specifically, according to the YOCO truncation requirements, for an input length L, the maximum hit length is at most L - 134 (window size 128 + speculative backoff 6). Under this upper limit, the Block KVCache hit status is checked first, assuming the hit length is L1 (L1 is an integer multiple of block size = 128), and then the SWA KVCache hit status is checked. If the SWA KVCache exists within the length range of L1 - 128 to L1, the hit length is L1; otherwise, the hit length is L1 - 128, where 128 is the recalculation window.

3FS Distributed File System

As mentioned above, Block KVCache handles massive amounts of data and relies on the 3FS distributed file system for storage. To cope with node downtime or network fluctuations, it needs to balance high availability with high performance.

• Data is stored on 3FS using erasure coding (10+2 Erasure Coding), and each EC group can tolerate the simultaneous failure of 2 nodes.
• During reading, requests are sent to all 11 Storage nodes in the group simultaneously, and any 10 responses are sufficient to decode and return the data.
• The GC Worker component is responsible for cleaning up expired caches: one instance is deployed on each Storage node, and the least accessed KVCache is periodically evicted according to the LRU policy, keeping disk usage below the waterline.

SWA Bounded Replay

SWA KVCache does not persist storage and recovers by recalculation when missing. This is the main change in KVCache management between V4 and V4. For details, please refer to the V4.1 technical report [2]. V4's approach is disk write plus precise recalculation: SWA KVCache writes to persistent storage in two locations. When a miss occurs, it recalculates "level number × window" tokens to perform a complete forward pass and precisely recovers the window state layer by layer. This path is very costly. It is necessary to use a sufficiently large SSD capacity to reduce the miss rate to an extremely low level. The cost is that SWA KVCache occupies nearly half of the capacity of persistent KVCache and does not bring corresponding hit benefits: Block KVCache is long-tailed and reused. A prefix can be repeatedly hit within several days. SWA KVCache is only reused within minutes in active sessions and becomes invalid when the session ends or enters the next round. Using SSD capacity that can be retained for several days to carry data with a lifespan of minutes is particularly uneconomical in multi-round short dialogue scenarios. Therefore, Zero SWA Caching (i.e., no disk writes and only precise recalculation) proposed in the V4 technical report was not actually adopted in the V4.1 inference system because its cost was too high for production deployment.

The V4.1 inference system employs SWA Bounded Replay, recalculating only the most recent 128 tokens and truncating the attention window within the recalculation segment: replaying begins from position s, and the query at position i is calculated only with the keys in the interval from max(s, i − 127) to i, resulting in an approximate SWA KVCache. The recalculation cost is thus reduced to a constant independent of the context length and number of layers. This mechanism is used in two places:

• Encoder SWA Bounded Replay: The prefix cache hit length now depends only on the Block KVCache. When the SWA KVCache is missing, the last 128 tokens of the hit prefix are recalculated and forwarded together with the missing suffix; the recalculated segment only rebuilds the SWA KVCache, while the Block KVCache is reused as is, without recalculation or overwriting.
• Decoder SWA Bounded Replay: Prefill computation is performed only at the encoder layer. In the CED architecture, the decoder layer's Block KVCache is projected from the encoder's last layer's hidden state. Only the decoder SWA KVCache needs to be calculated by each layer, requiring 20 decoder layers to process the last 2560 tokens for accurate reconstruction. Instead, each prefill operation only recalculates the last 128 tokens of the prompt at the decoder layer, and the output is only used for decoding and does not enter the prefix cache, nearly halving the total prefill computation.

DSpark Speculative Decoding

DSpark is a speculative decoding algorithm framework proposed by DeepSeek. Compared with DFlash, it superimposes a low-rank logit bias that only depends on the previous token ID on the output base logits. It uses extremely lightweight serial correction to fill in the local conditional information of adjacent tokens and improves the draft accuracy. For details, please refer to the DSpark paper [4].

At the inference system level, DSpark employs variable-length validation. Since the acceptance probability of draft tokens varies across different requests and locations, fixed-length validation wastes computing power on low-probability tokens. DSpark predicts the acceptance rate of each draft token using the Confidence Head and allocates validation computing power to the token with the highest marginal benefit during scheduling. In online deployment, a draft generates 5 tokens at a time. Based on the hardware throughput curve from the profile and the acceptance rate of each draft token for each request, the draft tokens are sorted according to their expected acceptance rates, and the subset of prefixes with the largest effective throughput is selected for validation. Comparison shows that compared to fixed validation using 5 tokens, variable-length validation improves throughput by 6% in online deployment scenarios.

In conjunction with Asynchronous Scheduling: When schedule(N+1) is executed, the confidence of step N has not yet been returned. Therefore, when the CPU schedules, only the total budget is determined. The request length is recalculated on the GPU by a Tilelang kernel (cumulative multiplication, global sorting to take the top-k, and fusion with remap scatter), and the result is written back to the pinned buffer.

Asynchronous Scheduling (Zero-Overhead Batch Scheduler)

The Nth step of the Decode process is: schedule(N) → forward(N) → sample(N) → post-process(N). Schedule and post-process are pure CPU logic. Before post-process(N), the GPU and CPU need to synchronously obtain the result of sample(N). This results in the GPU being idle during post-process(N) and schedule(N+1). Referring to existing implementations [5] [6], we overlap the pipeline through asynchronous scheduling between the GPU and the CPU: when the GPU executes forward(N) → sample(N), the CPU synchronously executes schedule(N+1) → launch_forward(N+1) → post-process(N-1) → launch_sample(N+1), and the GPU is not idle at all.

Structured output

Structured Output (SO) is used to constrain model output to strictly conform to a specified format (JSON schema, regular expression, EBNF, etc.).

Workflow:

• Compilation phase (independent SO machine): Compiles the format constraints into a syntax automaton.
• During runtime: At each decoding step, the automaton is updated, the set of valid tokens is determined, a mask is generated and applied to the sampling logic to force the output to conform to the target structure.

Compilation and derivation are decoupled; SO compilation is performed asynchronously, and the compilation results are retrieved before entering the engine.

References

1. ^ Y. Wu, S. Chen, R. Huang, Y. Tan, Y. Zhong, M. Zhang, X. Jin, and P. Huang. DualPath: Accelerating Agentic LLM Inference by Harvesting Disaggregated KV-Cache Storage I/O. In Proceedings of the ACM SIGCOMM 2026 Conference, SIGCOMM '26, Denver, CO, USA, 2026. ACM. ISBN 979-8-4007-2467-1.
2. ^ a b DeepSeek-AI. Deepseek-v4.1: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression, 2026. URL https://arxiv.org/pdf/2609.19969.
3. ^ DeepSeek-AI. A High-Performance Distributed File System Designed to Address the Challenges of AI Training and Inference Workloads, 2025. URL https://github.com/deepseek-ai/3FS.
4. ^ X. Cheng, X. Yu, C. Shao, J. Li, Y. Xiong, Y. Qian, J. Zhu, S. Ma, X. Zhang, J. Ye, et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv preprint arXiv:2607.05147, 2026a.
5. ^ L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, CH Yu, S. Cao, C. Kozyrakis, I. Stoica, JE Gonzalez, C. Barrett, and Y. Sheng. Sglang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems, volume 37, pages 62557–62583, 2024. doi: 10.52202/079017-2000.
6. ^ K. Zhu, Y. Gao, Y. Zhao, L. Zhao, G. Zuo, Y. Gu, D. Xie, T. Tang, Q. Xu, Z. Ye, K. Kamahori, C.-Y. Lin, Z. Wang, S. Wang, A. Krishnamurthy, and B. Kasikci. Nanoflow: Towards Optimal Large Language Model Serving Throughput. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation, OSDI '25, USA, 2025. USENIX Association. ISBN 978-1-939133-47-2.

Edited on 2026-10-20 04:30