Skip to content

Latest commit

 

History

History
140 lines (105 loc) · 15.8 KB

optimization-results.md

File metadata and controls

140 lines (105 loc) · 15.8 KB

Optimization results

Back to InferRSI · vLLM-Omni model optimization roadmap

The README highlights measured improvements of at least 20%. This page keeps the full curated record, including the campaigns below that bar. Hardware, workloads and baseline revisions differ across comparisons. Quality and acceptance status are reported for each result.

Text serving

DeepSeek V4.1 Flash W4A8 on Ascend A3

The project ran DeepSeek V4.1 Flash 1017 W4A8 on an Ascend A3 SuperPod with disaggregated prefill and decode (1P1D, 8 physical cards per side). AISBench generated load and InferRSI rechecked throughput at each scenario's latency limit. Project report, configuration and reproduction.

Scenario Per-card throughput metric Latency limit Baseline → optimized (tokens/s) Measured change
19 · Prefill Input throughput Mean TTFT < 10 s 68,239 → 105,397 +54.45%
21 · Decode, C1152 Steady-state output Mean TPOT < 50 ms 1,711.98 → 2,330.28 +36.12%
23 · End-to-end, steady state Decode output Mean TPOT < 21 ms 1,132.9 → 1,368.5 +20.80%

Cases 19 and 23 use medians from two runs. Case 21 uses one recheck of the retained configuration, which includes a SiLU safeguard. TTFT is time to first token. TPOT is time per output token.

Historical DeepSeek V4.1 Flash W8A8 pilot

The A3 / 910C pilot used 16 dies, TP4 / DP4 / EP16, DSpark-5 and an 8K-input / 1K-output workload. The slide deck reported the following single-run observations against the earlier control in issue #329.

Concurrency Earlier control → reported pilot TPOT (ms/token) Measured change
1 93.86 → 30.02 −68.0%
4 97.36 → 24.98 −74.3%
16 102.33 → 36.55 −64.3%

Only C4 reached 25 ms/token in the reported run. The follow-up benchmark uses its own calibrated baseline. The campaign stopped with an INCONCLUSIVE verdict. Follow-up #353.

Image, video and speech in vLLM-Omni

InferRSI campaigns and upstream experiments cover image, video and speech models. Each result applies to its linked workload and measured revisions. Some results are reported directly by upstream contributors, with validation scoped to their benchmarks.

Model Hardware Optimization approach Measured gains Validation and limits Evidence vLLM-Omni PR
Qwen-Image-2.1-Turbo 4× B300
  • Text-conditioning reuse
  • BF16 operator fusion
  • Symmetric-memory Ulysses with batched QKV exchange
384.16 → 266.60 ms (−30.60%)
Warm HTTP end-to-end latency, repeated prompts
Historical InferRSI campaign. Latencies are medians of five trial means from alternating AB/BA groups. All 44 RGBA images were pixel-identical. Campaign #3 · Pinned results #8739 (merged)
Qwen-Image-2.1-Turbo 1× Ascend 910B1
  • Conditioning cache for repeated prompts
  • Shared modulation once per denoising step
  • Native interleaved NPU RoPE with frequency-table reuse
2,921.23 → 2,272.83 ms (−22.20%) offline generation latency
Warm repeated prompts
Upstream parent-to-merge reference. InferRSI imported the techniques as knowledge. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle INCONCLUSIVE. Final-merge measurements · Earlier Ascend cohorts #8729 (merged)
MiniMax-H3 Ref2VA 8× Ascend 910B1
  • Lossy Cache-DiT profile tuning
407.27 → 108.12 s (−73.45%)
5 s video, HTTP end-to-end vs. dense inference
All 14 frozen quality comparisons passed. Campaign #293 · Pinned results #9698 (draft)
MiniMax-H3 Ref2VA 4× Ascend 950PR (~86 GB per card)
  • USP4 + MXFP8
  • RainFusion + Cache-DiT
701.95 → 166.17 s (−76.3%, 4.22×)
5 s clip denoising time vs. the in-run dense control
Five retained historical A/B slots. Performance and quality qualification remained inconclusive. For quality, the recipe reports sharpness, flicker, and luma ratios of 0.924 / 0.950 / 0.984 against a dense USP4 + MXFP8 reference. Archive record #520 · Measured repeats #9955 (merged recipe)
MiniMax-H3-Image 4× Ascend 910B1
  • Opt-in VAE tile graphs
  • Approximate encoder/decoder tile batching
16.89 → 10.82 s (−35.9%) · 4736×3552 generation
4.18 → 2.67 s (−36.1%) · 1024×1024 generation
Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed across generation and editing. Latency targets were unmet and final qualification remains INCONCLUSIVE. Campaign #390 · Evidence PR #394 #10184 (merged)
MiniMax-H3-Image 2× RTX 3090
  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
36.62 → 14.59 s (−60.16%)
Warm mean HTTP delivery latency
One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. Paired deployment benchmark #10243 (open)
MiniCPM-o 4.5 1× L20X Seed-TTS runtime and deployment changes 11.46 → 22.99 requests/s (+100.6%) and 0.65548 → 0.32300 mean RTF (−50.72%) at C32
10.62 → 18.51 requests/s (+74.29%) and 0.34287 → 0.19744 mean RTF (−42.42%) at C16
2.94 → 3.66 requests/s (+24.49%) at C1
Five retained fresh servers per arm, 12,800 successful requests. The independent-engine target was outside this comparison. Formal speech-quality non-inferiority remains unestablished. Campaign #431 #10331 (merged)
MiniCPM-o 4.5 1× L20X
  • Packed conditioning on a tuned configuration
  • TensorRT speaker extraction + CUDA MPS (separate comparison)
−27.69% median TTFA
Packed conditioning, C32, same 50-frame window
+20.5% at C16 · +25.4% at C32
TensorRT + MPS throughput
The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict FAIL. Campaign #49 #9218, #9188 (both merged)
LFM3-Omni-2B 1× Ascend 910B1
  • Depth graphs
  • Grouped greedy sampling
0.285 → 0.593 requests/s (+108.1%)
Computed from the rounded means; upstream quotes +108.4% from its unrounded values.
Upstream offline Omni API A/B, C4, temperature 0
Retained audio waveforms matched bit-for-bit within each tested temperature. Offline A/B benchmark #9827 (merged)
Drama 3 2× Ascend 910B1 Full graph replay for steady-state decode 17.31 → 22.40 / 22.91 frames/s (+31%) and 1.237 → 0.962 / 0.947 mean RTF (−23%) at C1 Rounded changes use the mean of the two candidate runs. The real-time factor targets were unmet. Campaign #132 #9308 (merged)
Qwen3-TTS 1.7B 1× Ascend 910B1 Codec and predictor optimizations 1.869 → 4.822 audio s/s (+158%) at C1
16.463 → 38.755 (+135%) at C16
21.245 → 52.091 (+145%) at C32
Historical seven-path profile vs. fresh same-card no-PR control. Mandarin speaker similarity and some ASR results regressed. Upstream reference. vLLM-Omni roadmap #416 #9068 (merged)

Earlier Qwen-Image-2.1-Turbo Ascend measurements. On one Ascend 910B1 at 1280×704, BF16, eager, and eight native steps, the author reported a 3111.5 ms baseline. Conditioning reuse with interleaved RoPE reached 2459.6 ms (−21.0%) with pixel-identical RGBA. Adding the opt-in VAE fastpath reached 2425.5 ms (−22.0%) and passed the reported quality gate with changed pixels. Each mean covers three requests after two warmups. InferRSI verification covered the import, so acceptance remains INCONCLUSIVE. The 30% target was unmet. The B300 campaign and this cohort used different hardware and measurement protocols. Earlier Ascend report

Historical campaigns below the 20% highlight bar

These campaigns stay on the record even though their measured gain is under the bar the README highlights. Their upstream PR labels are kept as they were.

Model Hardware Measured change Validation and limits Evidence vLLM-Omni PR
MiniMax-H3 Ref2VA 4 logical Ascend 910C devices 291.25 → 286.67 s (−1.57%)
5 s video, HTTP end-to-end
One lossless screening pair; strict quality checks passed for the screened candidate. Campaign INCONCLUSIVE. Campaign #297 · Evidence PR #317 #9745 (draft)
Qwen3-Omni 2× L20X +7.79% throughput · −6.06% mean RTF
One C64 pilot
Median time to first audio regressed 7.12%. Campaign INCONCLUSIVE. Campaign #112 · Pinned pilot #9283 (draft)
LingBot-World 4× Ascend 910C1 108.29 → 103.89 s (−4.06%)
Full-request latency, upstream AB/BA
Pre-codec tensors and decoded RGB frames matched exactly. KV gather benchmark #8112 (merged)

Historical MiniMax-H3 Ref2VA step-reduction measurements on 950PR

The author reported these 5-second workloads on four Ascend 950PR cards relative to their corresponding 50-step dense controls. Each sampling configuration used three seeds and eleven prompts. Both performance and quality qualification remained INCONCLUSIVE. One seed in the 10-step dense configuration lost 9.17% subject consistency. The 10-step Cache-DiT route combines fewer denoising steps and lossy caching, so its speedup describes the whole workload change. Record #520 · Evidence index #527.

Configuration Speedup Relative time reduction
10-step dense 5.04× −80.2%
10-step + Cache-DiT 0.12 7.37× −86.4%
11-step dense 4.74× −78.9%
12-step dense 4.34× −77.0%

TTFA is time to first audio. RTF is real-time factor. WER is word error rate.

Upstream PR statuses checked on 2026-11-20. To refresh a row, substitute its PR number in gh pr view 10184 --repo vllm-project/vllm-omni --json state,isDraft.

Standalone engine benchmarks

Standalone video/audio engines

h3-metal provides Metal-native FastH3 and MiniMax-H3 text-to-video/audio inference on Apple GPUs.

Author-reported results on Apple M5 Max (40-core GPU), 128 GB, macOS 27, at 768×448, 49 frames, seed 0, using real weights with one engine running at a time.

Engine Device Steps Cold Warm vs. MLX (cold / warm) vs. h3.c (cold / warm)
PipeNetwork/minimax-h3-mlx (4-bit) Apple M5 Max (40-core GPU) 20 153.1 s 134.6 s 1.00× / 1.00× 0.44× / 0.54×
antirez/h3.c, default preset Apple M5 Max (40-core GPU) 20 67.6 s 73.3 s 2.26× / 1.84× 1.00× / 1.00×
h3-metal, original H3 Apple M5 Max (40-core GPU) 20 54.29 s 29.89 s 2.82× / 4.50× — / 2.45×
h3-metal, FastH3 4-step + block-sparse attention Apple M5 Max (40-core GPU) 4 ¹ 48.23 s 18.20 s 3.17× / 7.40× 1.40× / 4.03×

Cold includes the first clip's weight loading, text encoding and preparation; warm reuses loaded weights and prepared state. h3-metal's cold time is 19.7% below h3.c's, just under the 20% bar this page highlights.

¹ FastH3 uses a distilled 4-step checkpoint, unlike the 20-step configurations. Warm values average clips 2–4 for the 20-step rows and two A/B/B/A warm runs for FastH3.

Author's report and reproduction →. These external results were not rerun in InferRSI.

Standalone speech engines

Concurrency 1 (C1) results reported by each project. Values are engine / vLLM-Omni / SGLang-Omni; compare within a row, since workloads and audio-frame definitions differ.

Project Device Throughput (frames/s) ↑ TTFA p50 (ms) ↓ Report and status
Drama 3 1× B300 309.17 / 195.89 / 169.80 32.83 / 47.29 / 79.38 3-run medians; overall target unmet
Qwen3-TTS 1× Ascend 910B1 61.12 / 12.71 / 4.31 50.41 / 263.51 / 439.72 3-run means; source campaign accepted
MiniCPM-o 4.5 1× L20X 397.43 / 187.52 / 278.72 126.2 / 295.3 / 381.2 5-run means; target unmet, quality unqualified

MiniCPM's SGLang baseline returns a complete waveform, so its TTFA equals end-to-end latency. These external results were not rerun in InferRSI.