Loading…
Research
lab358 serves the open-weight model you already use, exactly as its authors published it, and makes its memory durable: a document is processed once into a stored form and retrieved on demand rather than re-read on every call. There is no fixed context limit. Here is the published work we build on, what we integrate that is new, and how the approach differs from the common alternatives.
The idea
In a conventional stack, a long document is pushed through the prompt on every request and re-read from scratch — and the window it has to fit inside has a fixed ceiling. lab358 changes what the model is given: a document is processed once into stored memory, and approximate-nearest-neighbour search selects the parts relevant to the question at hand.
The approximation is in the selection, never in the arithmetic. Search decides which stored memory is put in front of the model; the model then attends over exactly that, and over recent context, merged into a single result — the same attention its authors shipped, unchanged. And because stored memory does not have to sit in GPU memory to be usable, how much a model can draw on is not decided by what fits alongside it on one device — hence no fixed context limit.
A document's key–value memory is computed once and reused wherever it is needed — the same precomputed memory serves every later request against it, with no re-encoding.
Lineage
lab358 is an integration of ideas from a rich body of published research. A selection of the work most directly related to how our attention and retrieval are built:
Liu, Chen, et al. — arXiv:2409.10516, 2024
Retrieves the most relevant key–value vectors per step from an approximate-nearest-neighbour index, and names the query-versus-key distribution mismatch that a retrieval index has to overcome.
Read the paper →Bertsch, et al. — NeurIPS 2023, arXiv:2305.01625
An early, clean statement of the core idea: let a model reach into an index for the entries it needs instead of holding everything in one window.
Read the paper →Mohtashami & Jaggi — NeurIPS 2023, arXiv:2305.16300
Makes block selection native to the attention mechanism itself rather than a bolted-on external retriever — the same philosophy of no fixed context window.
Read the paper →Tang, et al. — ICML 2024, arXiv:2406.10774
Estimates which key–value regions matter for the current query and loads only those — the query-aware selection principle at the heart of our index.
Read the paper →Subramanya, et al. — NeurIPS 2019
The graph-based nearest-neighbour algorithm that made billion-scale search practical on a single machine — the result that established retrieval at this scale as an engineering problem rather than a research one.
Read the paper →Chen, et al. — VLDB 2024, arXiv:2408.08933
Builds the search graph under the query distribution to fix recall on out-of-distribution queries — how we keep retrieval accurate when queries and stored keys differ.
Read the paper →A selection, not the full bibliography. Each ingredient above is well established in the literature; what lab358 contributes is the integration.
What's different
Each piece has strong prior art. Bringing them together into a single trained model and serving stack is where lab358 is distinctive.
We serve the open-weight model exactly as its authors published it. What changes is what the model is given to read: a document is processed once into stored memory and retrieved on demand, instead of being pushed through the prompt again on every call. No conversion, no retraining, no fine-tune — so the model's published evaluations remain its evaluations.
The nearest-neighbour graph is built to stay accurate for the queries attention actually issues, so the entries it returns are the ones the model would have chosen — closing the gap between approximate search and exact attention.
Stored memory does not have to sit in GPU memory to be usable, so how much a model can draw on is not decided by what fits alongside it on one device. Recent context stays immediately at hand; the rest is retrieved when it is wanted.
A document's key–value memory is computed a single time and reused across chats, agents, and users — no re-encoding and no prefill to redo, even days apart. Reuse is a first-class product primitive, not an accident of caching.
Approach comparison
A comparison of techniques, not products. Each row describes a way of giving a model access to more context, and where it runs into a wall.
The resolution
A document is processed once into stored memory. Approximate-nearest-neighbour search selects which of it is relevant to the question; the model then attends over exactly that, merged with recent context into one result.
The result: No fixed context window and no re-encoding: memory is retrieved into attention directly, and precomputed document memory is shared across uses.
The research, in a product you can run — hosted on lab358 Cloud or in your own AWS account.