Skip to content

[Dory] Enable prefix caching for per_loop and last_loop KV with SWA reclamation - #190

Open
HowardYangAI wants to merge 4 commits into
InferMatrixLab:mainfrom
HowardYangAI:dory-2c-prefix
Open

HowardYangAI wants to merge 4 commits into
InferMatrixLab:mainfrom
HowardYangAI:dory-2c-prefix

Conversation

@HowardYangAI

@HowardYangAI HowardYangAI commented Nov 16, 2026 •

Copy link
Copy Markdown

Required human contributor answers

1. What outcome are you trying to achieve?

Avoid repeating prefill computation for Dory requests with shared prefixes (scope).

2. What is the highest-leverage variable for that outcome?

The number of shared prompt tokens that are recomputed.

3. How will this work move that variable rather than create busywork?

Publish the KV each plane owns and restore it on a hit, so those prompt tokens can be skipped (reported failure and approach). This now covers both KV modes and works alongside SWA reclamation, so a hit no longer requires turning reclamation off.

4. Who is the customer or user, and what do they actually need?

Dory users sending requests with shared prompt prefixes need to avoid repeating that work, without giving up the memory that SWA reclamation saves.

PR necessity and bugfix evidence

A PR is needed: Dory rejects prefix caching, and removing that guard still leaves publication blocked because boundary layers only write plane 0 (failure report). With SWA reclamation, a hit could also fail admission, leave the reclamation frontier at zero, or find nothing to reuse because a long submission retired its prompt pages before they were published. Each of these was reproduced and has a regression test.

Purpose

Enable Dory prefix caching on grouped KV:

  • per_loop: cache boundary KV at plane 0 and recurrent KV at every plane.
  • last_loop: cache the shared plane in sync eager. The shared runner finishes every loop of a token before the next, so the cached final-loop KV is the cold state. CUDA Graphs and async scheduling still reject this combination.
  • SWA reclamation: a hit attaches only the cached pages still inside the window and starts reclaiming from there. Each step publishes its prompt pages before retiring expired ones.

Following review, reclamation is no longer a cache option. The engine picks the layout:

  • Grouped storage is used unless the execution needs all-layer blocks: CUDA Graphs, async scheduling, packed prefill, speculation, FP8 KV or preemption. Prefix caching works in both layouts.
  • Grouped storage reclaims expired SWA pages whenever exits are fixed-loop ouro, with incremental allocation or preallocation.
  • Automatic packed prefill ([perf][dory] add opt-in packed Triton prefill #189) still wins where a backend supports it, so SM120 BF16 defaults to all-layer blocks; packed_prefill=False keeps grouped storage.

Commits:

Commit Author Content
6b784bf Dragonborn 2B prerequisite: grouped hybrid KV storage and SWA reclamation
0f1f07c HowardYangAI per_loop prefix reuse
d8dfe33 HowardYangAI last_loop prefix reuse; prefix caching with SWA reclamation; rebase integration with FP8 KV (#129), packed prefill (#189) and Ascend (#168)
8e854d3 HowardYangAI Review changes: engine-derived reclamation, preallocated reclamation, admission budget for prefix hits, preemption fallback, required plan_cache layout arguments

The diff against main includes the 2B commit.

Follow-up, not in this PR: grouped storage could handle packed prefill and FP8 KV natively, by recording consumed positions for packed rows and splitting FP8 scales per layer. Then those two fallbacks would no longer be needed.

Addresses #176.

Test Plan

  • pre-commit run --all-files (Ruff lint, formatting and CPU regressions); standalone full CPU suite.
  • GPU suites for Dory, SWA reclamation, hybrid KV, FP8 KV, speculation, memory planning, backend selection and prefix growth, on SM89 and SM120.
  • Real-checkpoint cold/hit/branch parity on SM120 across nine configurations.

vllm-rlt Version / Commit: main f942c81, head 8e854d3.

Test Result

Check Result
Pre-commit: Ruff lint, format and CPU regressions Passed
Standalone full CPU suite 1364 passed, 927 skipped, 0 failed
GPU suites, L20 (SM89) 466 passed, 10 failed (unrelated, below)
GPU suites, RTX 5090 (SM120) 469 passed, 7 failed (unrelated, below)
Real-checkpoint prefix reuse, RTX 5090 9 configurations, all cold/hit/branch comparisons matched

Unrelated GPU failures:

  • 6 FlashAttention-4 cases in test_prefix_growth.py: the pinned flash-attn-4 release supports neither SM89 nor SM120. Same on main.
  • test_cuda_memory_plan_reclaims_allocator_residue_on_recreation: fails only when run with the full group, passes alone. Same on d8dfe33.
  • 3 FP8 store tests on L20 only. They also fail on main there, and pass on the 5090.

The packed prefix test (test_packed_prefill_prefix_hits_use_all_layer_blocks) needs the SM120 packed sliding-window tile; it ran and passed on the 5090.

GPU environments:

  • L20 (SM89): Python 3.12.3, PyTorch 2.13.0+cu129, Triton 3.7.1.
  • RTX 5090 32GB (SM120): PyTorch 2.12.1+cu130, Triton 3.7.1, driver 595.71.05.

Checkpoint from the ModelScope mirror, weights SHA-256 207483664dfef54c4c4438863260d3ba423bf3d6009b43fe84264b281e9fda23.

Real-checkpoint results

BF16, Triton, 2 loops, greedy, 16 generated tokens, 16-token blocks. Prompts are 1100 or 4096 tokens of repeated English text; the branch replaces the last 32 tokens. Each configuration runs cold, hit and branch with prefix caching on, and the same requests with it off as the control.

Config Storage Reclaim Tokens reused (hit / branch)
per_loop, 1100 tokens grouped yes 1088 / 1056
last_loop, 1100 tokens grouped yes 1088 / 1056
SWA, incremental, 4096 tokens, chunk 256 grouped yes 4080 / 4064
SWA, incremental, 4096 tokens, chunk 2048 grouped yes 4080 / 4064
SWA, preallocated, 4096 tokens grouped yes 4080 / 4064
Explicit packed prefill all-layer no 1088 / 1056
Automatic packed prefill all-layer no 1088 / 1056
FP8 E4M3 KV all-layer no 1088 / 1056
Preemption with CPU offload all-layer no 1088 / 1056

In every configuration, hit equals cold, branch equals the cache-off control, and all request-owned pages are released. FP8 is compared only with its own FP8 control. With a 1024-token window, the 4096-token hit starts reclaiming at page 191 and advances to page 193; the chunk-2048 run has one submission larger than the window. The preallocated run keeps a zero growth budget throughout prefill. The preemption run offloads the hit's KV to CPU after two tokens, resumes it once, and matches cold.

Time to first token, seconds, single runs:

Config Cache on: cold / hit Cache off: repeat
per_loop 0.899 / 0.103 0.870
last_loop 96.382 / 1.053 95.625
SWA incremental, chunk 256 4.011 / 0.113 4.035
SWA preallocated 3.944 / 0.109 3.922
Automatic packed 0.399 / 0.087 0.389

These are single runs and do not establish a speedup. last_loop sync eager prefills one token at a time through every loop, which is why its cold TTFT is high; a hit skips that work. Real-weight concurrency was 1 or 2; multi-request admission is covered by unit tests. BF16 can round differently when chunk shapes change, and the comparisons cover only the tested inputs.

Comment thread vllm_rlt/models/dory.py Outdated
# target's first loops, and boundary KV is committed by the target pass.
supports_self_speculation = True
# Complete-state prefix reuse: boundary KV is published from depth 0 and
# recurrent KV from every depth. Prefix caching requires per_loop KV.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we support prefix caching for last_loop in sync eager? The shared-KV runner finishes all loops for each token before starting the next, so restoring the final-loop KV preserves the history the next token reads. I confirmed identical cold and cache-hit logits on a small CPU run with the guard bypassed. Please add a regression test with prefill_chunk_size > 1.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. last_loop now supports prefix caching in sync eager; Graphs and async still reject it because they change the token order. Added regression tests with chunks 3/4 (CPU) and 3/16 (GPU). Real weights on an RTX 5090 also match the cold run.

Comment thread vllm_rlt/engine/stage_engine.py Outdated
if layer_windows is not None:
if scheduler_config.enable_preemption:
raise NotImplementedError("grouped KV does not support preemption snapshots")
if cache_config.enable_prefix_caching and cache_config.reclaim_swa:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we support prefix caching with SWA reclamation? Cache entries already hold their own page references, so we can resume from the last complete hit and recompute the rest. With the guard bypassed, an 8-token hit failed admission under a 4-token submission cap, and a shorter hit left the reclamation frontier at zero. Please handle both cases and test that reclamation continues after a hit.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both cases are fixed: admission now budgets each request's own peak, and a hit starts reclaiming from cached_tokens. I also found that a submission longer than the window could retire its prompt pages before publishing them, so publication now runs first. test_reclaim_continues_after_prefix_hit checks that reclamation keeps advancing after a hit; on the RTX 5090 a 4096-token hit reuses 4080 tokens and keeps reclaiming.

prettygirlisnotme and others added 3 commits November 20, 2026 15:31
Publish complete prefix pages using boundary and recurrent plane ownership,
retain cached pages after request release, and restore coverage on hits.
Use incremental prefix hashing and completion-aware publication, releasing
pending records with their allocation.

Keep compatible all-layer storage for graph, async and speculative execution,
and harden grouped allocation, padding, reclamation and initialization paths.
Preserve CacheConfig positional compatibility and document CPU/GPU validation.

This commit squashes only the 12 HowardYangAI commits. The prerequisite 2B
storage/reclamation change remains a separate commit by its original author.
Comment thread vllm_rlt/engine/stage_engine.py Outdated
)
layer_windows = None
if layer_windows is not None:
if scheduler_config.enable_preemption:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we preserve preemption by selecting the existing all-layer KV layout when KV offload is needed and grouped snapshots are unavailable? This guard rejects Dory sync eager even with prefix caching disabled. With all-layer KV, I verified a prefix hit followed by suspend and restore on a small CPU model with both features enabled. Slot-only preemption already keeps KV resident.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enable_preemption now selects all-layer KV instead of raising. Offload is decided at runtime, so the engine picks the layout up front. Added a test where a prefix hit is preempted, offloaded to CPU and restored; it matches the cold run on CPU and on the RTX 5090.

Comment thread vllm_rlt/config.py Outdated
# Admission headroom for incremental allocation; otherwise effective watermark is zero.
watermark_ratio: float = 0.0
# Ordinary fixed-loop grouped KV only; return expired SWA pages to the pool.
reclaim_swa: bool = False

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove reclaim_swa from CacheConfig and keep reclamation independent of incremental_allocation? Allocation policy controls when pages are acquired. Sliding-window reclamation controls when expired pages are safe to release, including pages allocated up front. The runtime can derive reclamation from the layer windows and execution capabilities, while memory planning accounts for the chosen allocation policy.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reclaim_swa is removed from CacheConfig. The engine enables it for grouped storage with fixed-loop ouro exits, and it no longer depends on incremental_allocation. Preallocation now reclaims too, and I fixed two budget issues that came up along the way: retired pages were counted as growth, and prefix hits were charged for expired pages. One trade-off: automatic packed prefill (#189) still takes priority, so SM120 BF16 doesn't reclaim by default (packed_prefill=False keeps grouped storage). Supporting packed and FP8 natively in grouped storage would remove that fallback; I'd suggest it as a follow-up.

Comment thread vllm_rlt/core/memory.py Outdated
# expose the compact count; everything else keeps the layer count.
kv_layers = getattr(config, "num_kv_layers", config.num_hidden_layers)
# Omitted uses the model policy; explicit None forces all-layer blocks.
if layer_windows is ...:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make layer_windows a required keyword argument and remove the Ellipsis default and model lookup. StageEngine already selects the KV layout and passes either the layer windows or None. plan_cache() should size the layout it receives. Tests can pass the intended layout explicitly.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. layer_windows and reclaim_swa are now required keywords, and tests pass the layout explicitly.

- Remove CacheConfig.reclaim_swa. The engine enables reclamation whenever
  grouped storage runs with fixed-loop ouro exits; other exit modes keep
  grouped pages resident.
- Decouple reclamation from incremental allocation. Preallocation reserves
  the full peak, still returns expired pages, and no longer budgets retired
  pages as growth.
- Charge a prefix hit only for the cached pages it attaches, so expired SWA
  pages stay evictable and do not delay admission.
- Select all-layer KV for preemption instead of raising, so suspend, CPU
  offload and restore work with prefix caching.
- Make plan_cache layer_windows and reclaim_swa required keywords chosen by
  the engine.
@HowardYangAI HowardYangAI changed the title [Dory] Enable per-loop prefix caching [Dory] Enable prefix caching for per_loop and last_loop KV with SWA reclamation Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants