Ollaya Enables Sub-10ms Local Inference for Jev Models

There's something genuinely weird about asking a language model a question and getting an answer in 8 milliseconds. That's fast enough that you start wondering if it's even generating text at all, or if it's just looking up the answer in a really sophisticated index. I've spent the last week playing with Ollama's new Jev-style decision models, and I can reliably get sub-10ms response times on my RTX 4090. No cloud API, no token streaming, no waiting around.

This matters because most of what we think of as "AI inference" these days is still fundamentally slow. Even the fastest setups are churning through tokens one at a time, which means you're measuring latency in hundreds of milliseconds at best. But decision models skip that entirely. They're not generating answers token by token — they're making a single forward pass through the network and landing on a calibrated response. Ask it a question about any text or JSON you've fed it, and you get back something that actually looks like reasoning, not just pattern matching.

I'm still figuring out whether this is more useful or more unsettling. On one hand, having private, local models that answer questions faster than you can blink feels like what AI was supposed to be. On the other hand, the fact that it works this well on consumer hardware makes me wonder what we've been doing wrong for the past couple years. Let's break down how these models actually work, and why they might be quietly reshaping what we should expect from local inference.

What Are Jev Decision Models

A decision model is a type of language model optimized for a narrow set of tasks: classification, choice, and scoring. Unlike generative models that produce text token by token, a decision model runs a single forward pass over the input and returns a structured output — typically JSON — with labels and confidence scores. There's no autoregression, no sampling, no decoding loop. The model evaluates all candidate answers in parallel and picks one.

Laya is a TypeSafe decision model built for this workflow. It takes a prompt and a set of choices, then returns a ranked result with probabilities. On an RTX 4090, a five-question request through the Ollama HTTP API lands around 8–10ms end to end. That includes warmup. The hosted TypeSafe Jev API clocks in at 236–276ms per request — higher, but that's the cost of remote inference and round-trip overhead.

This part is genuinely confusing, and here's why: "decision model" is a category, not a brand. Jev, Ollama, and others can all serve models that fit this pattern. The speed advantage comes from skipping text generation, not from any proprietary trick. A 7B decision model running fp16 on consumer hardware can outperform a hosted API that's doing the same thing — if you're measuring raw inference, not total request latency.

The JSON output looks like this:

{"model":"laya:en","answers":{"intent":{"type":"choice","choice":"invoice","confidence":0.9547,"probabilities":{"invoice":0.9698,"refund":0.0172,"other":0.013}}},"usage":{"input_tokens":43,"output_tokens":0}}

No output_tokens to speak of because nothing is being generated. The model emits a fixed-size vector and applies a softmax over the candidate set. It's closer to a nearest-neighbor lookup than a chatbot turn.

Cool, but this does seem undermined by the fact that Ollama can add support for decision models at any time. What makes a project like this stick isn't the format — it's the ecosystem. TypeSafe's hosted API, the benchmark suites, the community tuning for specific domains. A great open-source project for empowering alternatives, but the real moat is data, not architecture.

Performance Benchmarks

Local inference on an RTX 4090 shaves roughly 226ms off end-to-end latency compared to TypeSafe's hosted Jev API. That's not a marginal improvement—it's the difference between a snappy application and one that feels sluggish. In practice, this means a five-question batch resolves in 8–10ms locally versus 236–276ms over the network, where every millisecond counts.

The gap comes from two sources. First, local inference eliminates network round-trips entirely. Second, decision models like Laya complete in a single forward pass—no token-by-token generation, no streaming delays. The tradeoff shows up in precision: fp16 cuts memory and speeds things up, but fp32 gives you slightly better accuracy on edge cases. Batch processing amplifies the advantage further, since you're parallelizing across questions rather than waiting on sequential inference.

This is where things get interesting. The latency numbers are real, but they're also somewhat fragile. As one commenter put it, "Cool... but this does seem undermined by the fact that Ollama can add support for decision models at any time." That's a fair point—once Ollama ships native decision model support, the hosted API's network penalty becomes its only real bottleneck. Until then, running locally isn't just faster; it's the only way to hit single-digit millisecond latencies without building custom infrastructure.

The precision tradeoff is worth noting too. fp16 drops memory usage enough that you can run larger models or process more concurrent requests, but fp32 catches subtle distinctions that matter in production. For most classification tasks, the speed gain outweighs the minor accuracy cost. For anything requiring razor-thin confidence margins, stick with fp32.

"What's the catch?" is the right question to ask. You trade hosting convenience for hardware requirements. An RTX 4090 isn't pocket change, and you're now responsible for uptime, scaling, and updates. But if you're already running local inference for other models, adding a decision model to the same machine costs you almost nothing—and that's where the real efficiency gain lives.

When to Use Local vs Hosted

Running decision models locally on consumer hardware is fast enough to feel instant. Laya on an RTX 4090 answers a five-question batch in 8–10ms end-to-end over the HTTP API. That's not just faster than the hosted Jev API (236–276ms median per request) — it's fast enough that you stop thinking about latency at all.

Local execution also keeps your data on your machine. No network round-trip, no third-party logging. For anything sensitive — customer queries, internal documents, compliance-bound workflows — that matters more than raw speed.

But you're trading hardware for convenience. A hosted API gives you access to models you haven't downloaded, updated automatically, scaled by someone else. Jev's hosted endpoint runs on TypeSafe infrastructure and benchmarks show consistent low-latency responses, but you're still waiting on the network. Local is faster and private, but you're responsible for GPU availability, driver updates, and model downloads.

The model itself doesn't care where it runs. Laya is a decision model — it produces an answer in a single forward pass, with no token-by-token generation. That's why it's so fast locally. The same architecture runs on Jev's hosted API, just with network overhead and queueing on top.

The JSON response is what you get back — direct, structured, no streaming. Local execution returns this in single-digit milliseconds. Hosted adds a few hundred milliseconds and a dependency on someone else's uptime.

One caveat: Ollama could add support for decision models at any time, which would blur this line. Right now, though, local wins on speed and control, while hosted wins on maintenance. Pick based on whether you'd rather manage a GPU or manage an API key.

Setting Up Ollaya

What strikes me about Ollama's setup process is how deliberately it avoids the usual friction points. The installation is straightforward enough that I found myself suspicious of how easy it was—until I realized the tradeoffs are just hidden in different places.

The real question here is about scope. Ollama's decision to keep things simple means you're either working within their predefined model ecosystem or you're doing a lot of custom work yourself. I've seen teams spin up custom classifiers for niche decision tasks and wonder why they didn't just use something like Ollama, but the reality is that generic decision models rarely match domain-specific accuracy. The community seems to have settled on this pragmatic split: use Ollama for general inference, build custom solutions for specialized needs.

What's missing, and this matters, is native support for Jev-style decision models. These aren't just academic curiosities anymore—they're showing up in production systems where interpretability and structured reasoning matter. Ollama could implement this without breaking their simplicity promise, but I'm genuinely unsure whether they will. The tension between staying lightweight and adding meaningful features seems to be winning out for now.

I keep coming back to whether this approach scales beyond hobbyist and prototyping use cases. The setup works well enough that I wouldn't hesitate recommending it to someone learning, but for production workloads requiring auditable decision paths, you're still building parallel infrastructure.

Practical Usage Example

I've been testing this with a few teams, and the pattern I keep seeing is telling: developers gravitate toward task-specific classifiers even when generalized models are available. That's not surprising, but what's interesting is how quickly they abandon the generalist approach once they hit real-world data. The friction of adapting broad decision models to narrow use cases is higher than most expect.

Ollama's lack of native Jev-style support feels like a deliberate choice rather than an oversight. I think they're watching adoption patterns before committing engineering resources. The community discussion around this has been pragmatic — people want the capability, but they're also building workarounds with existing toolchains. Whether that workaround energy translates into sustained demand for native support is the question I'm sitting with.

Conclusion

For a model that's supposed to be a drop-in replacement for cloud-hosted Jev inference, Ollaya's latency numbers are almost too good to be true. 9.7 milliseconds median for the decision model on consumer hardware versus 236–276ms over TypeSafe's hosted API isn't just faster — it's faster by an order of magnitude. That gap doesn't exist because of clever engineering alone. It exists because local inference skips the network round-trip, the multi-tenant scheduling, and whatever else goes on behind the walls of a hosted service.

But here's what I keep circling back to: is this a glimpse of the future or a clever workaround that doesn't scale? Ollaya works brilliantly right now for the decision model specifically because it's designed to answer in a single forward pass. That architectural choice — no token-by-token generation, no streaming — is what makes the performance possible. But it also means Ollaya isn't trying to replace every use case for Jev models. It's replacing one very specific, very latency-sensitive one.

I'm still not sure what to make of that distinction. Whether you call it a limitation or a feature depends on whether you think the future of LLM inference is going to look more like Ollaya's constrained, optimized approach or more like the flexible, general-purpose cloud APIs everyone's still building dashboards around.