inference economics
Training gets the headlines, but the money is in inference — the large majority of AI compute spend, paid again on every request. And the workloads growing fastest are exactly the ones standard attention punishes most.
Training gets the headlines, but the money is in inference — the large majority of AI compute spend, paid again on every request. And the workloads growing fastest are exactly the ones standard attention punishes most.
The story everyone tells about AI cost is a training story: the enormous one-time run, the cluster, the months. It's a real cost — but it's the wrong one to budget around. Training a model is something you do once. Inference — running the model — is something you do on every single request, forever. That's where the spend actually accumulates.
Industry estimates bear this out. Inference is projected to grow from about half of all AI compute in 2025 to roughly two-thirds in 2026 (Deloitte), and it can reach 80–90% of the lifetime cost of a production AI system, precisely because it runs on every request rather than once. The AI inference market alone is estimated near $106B in 2025 and projected toward $255B by 2030 (MarketsandMarkets). The one-time cost gets the attention; the recurring one gets the invoice.
Here's the part that should worry anyone forecasting an AI budget: the fastest-growing inference workloads are the ones standard attention handles least efficiently — the long-document ones. Contracts. Case law. Patient histories. Regulatory filings. Whole codebases.
Standard attention makes these expensive by construction. Its cost grows with the square of the context length, and the key–value cache it builds devours GPU memory. Double the document, roughly quadruple the attention cost. So the workloads people most want to point AI at — "read this entire thing and reason over it" — are precisely the ones that scale the worst on the bill. Demand is climbing fastest where the unit economics are worst.
That gap is the whole game. If every request keeps paying full price to re-read the same documents, the most valuable AI workloads stay the most expensive ones, indefinitely.
The fix isn't a bigger GPU or a cheaper token — it's not paying for the same context twice.
We wrote last week about treating a document's context as a durable asset: index it once, reuse it across every chat, agent, and API call, with no prefill to redo. That's the reuse half of the story, and it's the larger half — the most expensive part of a long-document request is reading the document, and in a standard stack you pay it again on every call. The other half is what happens once a corpus is indexed: the model doesn't have to read all of it to answer a question. lab358 retrieves the relevant parts with approximate nearest-neighbor search, so growing the corpus doesn't grow the cost of each request in lockstep with it.
The model itself is untouched through all of this — the open-weight checkpoint you already use, served exactly as its authors published it. The work is in the serving layer, not in a modified model you'd have to re-evaluate.
Put the two together — don't re-read what you already indexed, and don't read what isn't relevant — and the long-document workloads that break a standard inference budget become the ones that hold up best under it.
lab358 is coming two ways — the same model and the same OpenAI-compatible API either way:
If long-context inference is a line item you're trying to forecast, join the lab358 Cloud waitlist — or talk to us about self-hosting.
Investors: hello@lab358.ai.