DeepSeek-V4 Architecture Overview

DeepSeek-V4 introduces a new generation of MoE-based models with optimized performance and efficiency. The architecture includes two variants: DeepSeek-V4-Flash 284B and DeepSeek-V4-Pro 1.6T, both designed to activate only a fraction of parameters during inference. Key innovations include hybrid attention mechanisms (CSA+HCA) for reduced long-context computational cost, mHC structures for network stability, and the Muon optimizer for improved training convergence.

GPUStack Platform Setup

GPUStack is an open-source platform for managing GPU clusters and deploying AI models. It supports multiple inference backends including vLLM, SGLang, and TensorRT-LLM, providing capabilities for orchestration, performance optimization, and enterprise-grade operations.

Container Environment Preparation

Ensure Docker is installed and running on all nodes:

docker info

Starting GPUStack Server

Deploy the GPUStack server container:

sudo docker run -d --name gpustack \
  --restart unless-stopped \
  -p 80:80 \
  -v gpustack-data:/var/lib/gpustack \
  swr.cn-south-1.myhuaweicloud.com/gpustack/gpustack:v2.1.2 \
  --debug --bootstrap-password GPUStack@123

Adding NVIDIA GPU Worker Nodes

Verify driver version and NVIDIA Container Toolkit configuration:

nvidia-smi
sudo docker info 2>/dev/null | grep -q "Runtime.*nvidia" && echo "NVIDIA Container Toolkit OK" || echo "Configuration required"

Register worker nodes using the command provided in the GPUStack console and verify their status.

Custom vLLM Configuration

Add a custom vLLM version compatible with DeepSeek-V4:

backend_name: vLLM
version_configs:
  0.20.0-cu130-custom:
    image_name: vllm/vllm-openai:v0.20.0-cu130
    entrypoint: vllm serve
    run_command: >-
      {{model_path}} --host {{worker_ip}} --port {{port}} --served-model-name
      {{model_name}}
    env: {}
    custom_framework: cuda

Deploying DeepSeek-V4-Pro

Deploy the model using either online (ModelScope) or offline (local weights) methods. Configure vLLM with the following parameters for optimal performance on 8x H200/H20 GPUs:

--trust-remote-code
--kv-cache-dtype fp8
--block-size 256
--enable-expert-parallel
--tensor-parallel-size 8
--max-num-seqs 512
--max-num-batched-tokens 512
--no-enable-flashinfer-autotune
--compilation-config '{"mode": 0, "cudagraph_mode": "FULL_DECODE_ONLY"}'
--gpu-memory-utilization 0.95
--max-model-len auto
--tokenizer-mode deepseek_v4
--tool-call-parser deepseek_v4
--enable-auto-tool-choice
--reasoning-parser deepseek_v4
--speculative_config '{"method":"mtp","num_speculative_tokens":1}'

Set environment variable: VLLM_ENGINE_READY_TIMEOUT_S=3600.

Performance Testing Results

Benchmark tests conducted with different configurations:

Throughput (Tokens/s)

Configuration	Throughput
TP 8 + EP, max-num-batched-tokens 512	2483.19
TP 8 + EP, max-num-batched-tokens 8192	6936.57
DP 8 + EP, max-num-batched-tokens 512	10289.27
DP 8 + EP, max-num-batched-tokens 8192	OOM

Latency (ms)

Configuration	TTFT	TPOT
TP 8 + EP	240.86	21.89
DP 8 + EP	273.45	33.21

API Integration Example

Test tool calling functionality with the OpenAI client:

from openai import OpenAI
import json

client = OpenAI(
    base_url="http://YOUR_SERVER_IP/v1",
    api_key="YOUR_API_KEY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="DeepSeek-V4-Pro",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=tools,
    max_tokens=1024
)

Production Optimizaton Recommendations

• Choose TP for lower latency per request, DP for higher overall throughput
• Adjust max-num-batched-tokens carefully to balance throughput and stability
• Limit context length to manage KV Cache memory usage
• Consider extended KV Cache solutions (LMCache/HiCache) for memory-intensive wrokloads
• Test MTP speculative decoding once compatible versions are available
• Prioritize stability over peak performance in production environments