durable context
In a standard stack the KV cache is scratch space — built during a request, sized by that request, thrown away at the end, and competing with the model for the same scarce memory. Make it durable instead and long context stops being a memory-allocation question.
In a standard stack the KV cache is scratch space — built during a request, sized by that request, thrown away at the end, and competing with the model for the same scarce memory. Make it durable instead, and long context stops being a memory-allocation question.
Ask why long context is expensive and you usually get an answer about compute: attention costs more as the input grows. That is true, and it is not the part that bites first in production.
The part that bites first is memory. When a model reads a document, it builds an internal working representation of everything it has read — the KV cache. That cache lives in GPU memory, next to the model weights, and it grows with every token. On a long document it stops being a footnote and starts being the largest thing on the device.
Which produces the awkward arithmetic every team running long context eventually does: the model needs memory, the context needs memory, concurrent users each need their own context, and they all need it from the same card.
Here is the thing worth noticing about that cache: in a standard stack it is transient. It is created when the request starts and discarded when the request ends.
Say that out loud and it sounds wasteful, because it is. The cache is the expensive part of a long-document request — it is what all that reading produced — and it is thrown away the moment the answer is returned. Ask a second question about the same document and the work is done again from nothing. Ask tomorrow, same. Ten people on the same contract, ten times.
The industry's usual response is prompt caching or prefix caching, and it helps, but notice what kind of help it is. It is opportunistic: it works when the next request happens to start with the same tokens, it is keyed to an exact prefix, and it expires on a timer measured in minutes. It makes the waste less likely. It does not change what the cache is.
Two consequences follow from the cache being scratch space, and they are the two complaints:
lab358 treats that representation as a durable artifact rather than a by-product. A document is processed once into a stored form the model can read back later, and retrieved on demand — with approximate nearest-neighbour search — instead of being pushed through the prompt again on every call.
The model itself is untouched. We serve the open-weight checkpoint you already use exactly as its authors published it: same weights, same tokenizer, same behaviour the community measured. What changes is the serving layer around it.
Once the cache is an artifact instead of scratch space, three things move at once:
The ceiling stops being set by one device. Stored context does not have to sit in GPU memory to be usable, so how much you can work with is no longer decided by what fits next to the model.
The expensive part is paid once. Reading the document is the cost. Index it once and the next turn, the next agent step, and the next user tomorrow all skip it.
Concurrency stops trading against context. This is the one people don't expect, and it is the one operators feel. When context is transient, every simultaneous user is holding their own copy in the same memory — so serving more people at once means giving each of them less room to read. When the context is stored and shared, that trade-off is not there to make. Ten people reading the same contract are reading one indexed copy.
It is worth being precise about the distinction, because the words are close and the ideas are not.
A cache is an optimisation. It is allowed to miss, it is allowed to expire, and nothing breaks if it does — you just pay full price again. That is exactly right for a cache, and it is why prefix caching is measured in hit rates.
A durable artifact is something you have. It is created on purpose, it is addressable, it does not expire out from under you, and it is reusable across chats, across agents, across users, across days — with no requirement that the next request happen to begin with the same tokens. You index a document because you intend to use it, not because you are hoping to get lucky on a prefix match.
That is the difference between making the waste less likely and not doing the work twice.
Two honest limits, because a post like this is easy to write without them.
The first read still costs what it costs. Indexing a document is real work on real hardware. The claim is that it is paid once, not that it is free.
Retrieval has to be good. Reading the relevant parts instead of all of them is only an advantage if the relevant parts are the ones that come back. That is a quality question, it is measurable, and it is ours to answer with numbers rather than adjectives — which is why you will see benchmark posts here with the model, the eval set, and the method named, and not before.
The reason to care about this framing is that it moves long context out of the category of hardware limit you wait for someone else to raise and into the category of system design. Context length stops being a spec-sheet number and starts being a property of how documents are stored and retrieved — which is a thing that can be built, measured, and improved without anyone shipping a bigger card or anyone retraining a model.
That is the bet lab358 is making, and the rest of the engineering follows from it.
Join the waitlist to try durable long context on a model you already use — managed on lab358 Cloud, or self-hosted in your own AWS account. Or talk to us if you want to walk through the fit for a specific workload.