Repository navigation
[Dory] Enable prefix caching for per_loop and last_loop KV with SWA reclamation - #190
HowardYangAI wants to merge 4 commits into
Conversation
| # target's first loops, and boundary KV is committed by the target pass. | ||
| supports_self_speculation = True | ||
| # Complete-state prefix reuse: boundary KV is published from depth 0 and | ||
| # recurrent KV from every depth. Prefix caching requires per_loop KV. |
There was a problem hiding this comment.
Can we support prefix caching for last_loop in sync eager? The shared-KV runner finishes all loops for each token before starting the next, so restoring the final-loop KV preserves the history the next token reads. I confirmed identical cold and cache-hit logits on a small CPU run with the guard bypassed. Please add a regression test with prefill_chunk_size > 1.
There was a problem hiding this comment.
Done. last_loop now supports prefix caching in sync eager; Graphs and async still reject it because they change the token order. Added regression tests with chunks 3/4 (CPU) and 3/16 (GPU). Real weights on an RTX 5090 also match the cold run.
| if layer_windows is not None: | ||
| if scheduler_config.enable_preemption: | ||
| raise NotImplementedError("grouped KV does not support preemption snapshots") | ||
| if cache_config.enable_prefix_caching and cache_config.reclaim_swa: |
There was a problem hiding this comment.
Can we support prefix caching with SWA reclamation? Cache entries already hold their own page references, so we can resume from the last complete hit and recompute the rest. With the guard bypassed, an 8-token hit failed admission under a 4-token submission cap, and a shorter hit left the reclamation frontier at zero. Please handle both cases and test that reclamation continues after a hit.
There was a problem hiding this comment.
Both cases are fixed: admission now budgets each request's own peak, and a hit starts reclaiming from cached_tokens. I also found that a submission longer than the window could retire its prompt pages before publishing them, so publication now runs first. test_reclaim_continues_after_prefix_hit checks that reclamation keeps advancing after a hit; on the RTX 5090 a 4096-token hit reuses 4080 tokens and keeps reclaiming.
Publish complete prefix pages using boundary and recurrent plane ownership, retain cached pages after request release, and restore coverage on hits. Use incremental prefix hashing and completion-aware publication, releasing pending records with their allocation. Keep compatible all-layer storage for graph, async and speculative execution, and harden grouped allocation, padding, reclamation and initialization paths. Preserve CacheConfig positional compatibility and document CPU/GPU validation. This commit squashes only the 12 HowardYangAI commits. The prerequisite 2B storage/reclamation change remains a separate commit by its original author.
| ) | ||
| layer_windows = None | ||
| if layer_windows is not None: | ||
| if scheduler_config.enable_preemption: |
There was a problem hiding this comment.
Can we preserve preemption by selecting the existing all-layer KV layout when KV offload is needed and grouped snapshots are unavailable? This guard rejects Dory sync eager even with prefix caching disabled. With all-layer KV, I verified a prefix hit followed by suspend and restore on a small CPU model with both features enabled. Slot-only preemption already keeps KV resident.
There was a problem hiding this comment.
enable_preemption now selects all-layer KV instead of raising. Offload is decided at runtime, so the engine picks the layout up front. Added a test where a prefix hit is preempted, offloaded to CPU and restored; it matches the cold run on CPU and on the RTX 5090.
| # Admission headroom for incremental allocation; otherwise effective watermark is zero. | ||
| watermark_ratio: float = 0.0 | ||
| # Ordinary fixed-loop grouped KV only; return expired SWA pages to the pool. | ||
| reclaim_swa: bool = False |
There was a problem hiding this comment.
Can we remove reclaim_swa from CacheConfig and keep reclamation independent of incremental_allocation? Allocation policy controls when pages are acquired. Sliding-window reclamation controls when expired pages are safe to release, including pages allocated up front. The runtime can derive reclamation from the layer windows and execution capabilities, while memory planning accounts for the chosen allocation policy.
There was a problem hiding this comment.
reclaim_swa is removed from CacheConfig. The engine enables it for grouped storage with fixed-loop ouro exits, and it no longer depends on incremental_allocation. Preallocation now reclaims too, and I fixed two budget issues that came up along the way: retired pages were counted as growth, and prefix hits were charged for expired pages. One trade-off: automatic packed prefill (#189) still takes priority, so SM120 BF16 doesn't reclaim by default (packed_prefill=False keeps grouped storage). Supporting packed and FP8 natively in grouped storage would remove that fallback; I'd suggest it as a follow-up.
| # expose the compact count; everything else keeps the layer count. | ||
| kv_layers = getattr(config, "num_kv_layers", config.num_hidden_layers) | ||
| # Omitted uses the model policy; explicit None forces all-layer blocks. | ||
| if layer_windows is ...: |
There was a problem hiding this comment.
Please make layer_windows a required keyword argument and remove the Ellipsis default and model lookup. StageEngine already selects the KV layout and passes either the layer windows or None. plan_cache() should size the layout it receives. Tests can pass the intended layout explicitly.
There was a problem hiding this comment.
Done. layer_windows and reclaim_swa are now required keywords, and tests pass the layout explicitly.
- Remove CacheConfig.reclaim_swa. The engine enables reclamation whenever grouped storage runs with fixed-loop ouro exits; other exit modes keep grouped pages resident. - Decouple reclamation from incremental allocation. Preallocation reserves the full peak, still returns expired pages, and no longer budgets retired pages as growth. - Charge a prefix hit only for the cached pages it attaches, so expired SWA pages stay evictable and do not delay admission. - Select all-layer KV for preemption instead of raising, so suspend, CPU offload and restore work with prefix caching. - Make plan_cache layer_windows and reclaim_swa required keywords chosen by the engine.
0837c45 to
8e854d3
Compare
Required human contributor answers
1. What outcome are you trying to achieve?
Avoid repeating prefill computation for Dory requests with shared prefixes (scope).
2. What is the highest-leverage variable for that outcome?
The number of shared prompt tokens that are recomputed.
3. How will this work move that variable rather than create busywork?
Publish the KV each plane owns and restore it on a hit, so those prompt tokens can be skipped (reported failure and approach). This now covers both KV modes and works alongside SWA reclamation, so a hit no longer requires turning reclamation off.
4. Who is the customer or user, and what do they actually need?
Dory users sending requests with shared prompt prefixes need to avoid repeating that work, without giving up the memory that SWA reclamation saves.
PR necessity and bugfix evidence
A PR is needed: Dory rejects prefix caching, and removing that guard still leaves publication blocked because boundary layers only write plane 0 (failure report). With SWA reclamation, a hit could also fail admission, leave the reclamation frontier at zero, or find nothing to reuse because a long submission retired its prompt pages before they were published. Each of these was reproduced and has a regression test.
Purpose
Enable Dory prefix caching on grouped KV:
Following review, reclamation is no longer a cache option. The engine picks the layout:
ouro, with incremental allocation or preallocation.packed_prefill=Falsekeeps grouped storage.Commits:
6b784bf0f1f07cd8dfe338e854d3plan_cachelayout argumentsThe diff against main includes the 2B commit.
Follow-up, not in this PR: grouped storage could handle packed prefill and FP8 KV natively, by recording consumed positions for packed rows and splitting FP8 scales per layer. Then those two fallbacks would no longer be needed.
Addresses #176.
Test Plan
pre-commit run --all-files(Ruff lint, formatting and CPU regressions); standalone full CPU suite.vllm-rlt Version / Commit: main
f942c81, head8e854d3.Test Result
Unrelated GPU failures:
test_prefix_growth.py: the pinned flash-attn-4 release supports neither SM89 nor SM120. Same on main.test_cuda_memory_plan_reclaims_allocator_residue_on_recreation: fails only when run with the full group, passes alone. Same ond8dfe33.The packed prefix test (
test_packed_prefill_prefix_hits_use_all_layer_blocks) needs the SM120 packed sliding-window tile; it ran and passed on the 5090.GPU environments:
Checkpoint from the ModelScope mirror, weights SHA-256
207483664dfef54c4c4438863260d3ba423bf3d6009b43fe84264b281e9fda23.Real-checkpoint results
BF16, Triton, 2 loops, greedy, 16 generated tokens, 16-token blocks. Prompts are 1100 or 4096 tokens of repeated English text; the branch replaces the last 32 tokens. Each configuration runs cold, hit and branch with prefix caching on, and the same requests with it off as the control.
In every configuration, hit equals cold, branch equals the cache-off control, and all request-owned pages are released. FP8 is compared only with its own FP8 control. With a 1024-token window, the 4096-token hit starts reclaiming at page 191 and advances to page 193; the chunk-2048 run has one submission larger than the window. The preallocated run keeps a zero growth budget throughout prefill. The preemption run offloads the hit's KV to CPU after two tokens, resumes it once, and matches cold.
Time to first token, seconds, single runs:
These are single runs and do not establish a speedup. last_loop sync eager prefills one token at a time through every loop, which is why its cold TTFT is high; a hit skips that work. Real-weight concurrency was 1 or 2; multi-request admission is covered by unit tests. BF16 can round differently when chunk shapes change, and the comparisons cover only the tested inputs.