Your model holds 128,000 tokens. Each step, it reads about 2,000.
Qache is an attention backend that stops paying to keep the other 98% in GPU memory. It indexes KV vectors off-GPU and retrieves only the keys a step actually attends to — using an attention-aware nearest-neighbour search built for the query/key distribution mismatch that makes off-the-shelf indexes miss.
Long context is priced like dense attention and used like sparse attention.
A 128K-token request pins gigabytes of key-value cache to HBM for its whole lifetime. Attention cost grows with every token you support, and the cache is what caps how many concurrent long-context requests fit on a card.
But attention is dynamically sparse. At any decoding step a head sends nearly all of its weight to a small, shifting set of keys. The rest of the cache is resident, paid for, and idle.
Four stages, once per request and once per step.
Prefill as usual
The model computes keys and values for the prompt. Nothing about the model changes — Qache is training-free and needs no fine-tuning.
Index off-GPU
KV vectors move to host memory and get an ANN index. A small set of hot keys stays on the card so common patterns never round-trip.
Retrieve per step
Each query vector searches the index. Queries and keys don't share a distribution, so generic ANN recall collapses here; Qache's search adapts to the query side instead of assuming symmetry.
Attend to what came back
Attention runs over the retrieved set plus the resident keys. Output stays within noise of full attention on long-context benchmarks.
Two changes worth planning around.
Serving density stops tracking context length
Cache size per request falls to the resident set plus an index that lives on cheap memory. More concurrent long-context requests per GPU, or the same traffic on smaller cards.
Context becomes a durable asset
An index outlives the request that built it. A repository or document corpus can be indexed once and attached to any later request without re-prefilling it — context you mount rather than context you rebuild.
Retrieval and long context stop being alternatives
RAG retrieves over chunks; this retrieves over KV states. Same operation, different granularity — and one system can now serve both ends of it.
Cases to route around, not oversell.
- Dense aggregationWorkloads that genuinely read most of the context — full-document summarisation, exhaustive extraction — break the sparsity assumption. Sparsity is a property of the workload, not a guarantee.
- Tail latencyHost-side retrieval on the decode path adds variance. Steady-state throughput improves; p99 needs its own budget and measurement.
- Short contextsBelow roughly 16K tokens the index build isn't repaid. Qache is a long-context path, and the router should treat it as one.
- Index constructionPrefill still happens, and building the index costs time proportional to context. It pays back across many decode steps, not on the first one.
What's measured, and what isn't yet.
| Measure | Full attention | Qache | Source |
|---|---|---|---|
| Share of KV cache read per step | 100% | 1–3% | Published method |
| 128K context, 8B model | Multi-GPU | Single 24GB card | Published method |
| Decode latency, 128K | — | 0.188 s / token | Published method |
| Long-context accuracy | Baseline | Within noise | Published method |
| Concurrent requests per GPU | — | Benchmark pending | Qache prototype |
| p99 decode latency under load | — | Benchmark pending | Qache prototype |
| Index build time per 100K tokens | — | Benchmark pending | Qache prototype |
Rows marked Published method are results reported for the approach Qache implements (Liu et al., RetrievalAttention, arXiv:2409.10516). Rows marked Qache prototype are ours to measure and publish before we claim them.