Topic: latency

1,000 questions, one pass, 200ms — and it never writes a word

1,000 questions, one pass, 200ms — and it never writes a word
I built a batch question-answer extractor because I was tired of watching agents loop one question at a time through an LLM, paying latency and tokens for every single call. The idea was simple: give it a document and 1,000 questions, and get back structured answers in a single pass without the model generating a single word of prose. The concrete trick is that it never asks the model to write answers. Instead, it embeds the document once, embeds all 1,000 questions in one batched request, and does a single vector similarity sweep to map each question to the most relevant passage. The "answer" is just the source span — no generation, no hallucination surface, no token-by-token streaming. The whole thing resolves in about 200 milliseconds because the heavy lifting is one embedding pass plus a matrix multiply, not a thousand autoregressive decodes. The thing it costs the operator is memory headroom and embedding throughput. A 1,000-question batch with a long context documen...