I built an independent clock to time Gemini 4 Hydrogen — the 2.5x claim vs my 1.1x, and what it cost me

Vendor decks love a multiplier. When the "Gemini 4 Hydrogen" numbers came out claiming 2.5x faster inference, I didn't take the marketing at face value, so I built a small harness that times every request with a monotonic wall clock around the full round trip — queue time, prefill, decode, everything — rather than the cherry-picked decode-only window the claim was based on.

When I ran the same 200-prompt set through my harness, the honest number came out at 1.1x. Not because the model is bad, but because the 2.5x only exists if you exclude everything a real operator pays for. I put the whole comparison into a video walkthrough so the methodology is visible instead of buried in a footnote: https://www.youtube.com/watch?v=DsOdtrugezg

One concrete detail: my clock starts before the request hits the API gateway and stops after the final token is parsed client-side, and I log both timestamps per request so I can show exactly where the time goes.

The thing it costs the operator is GPU-hours. Running a credible benchmark like this ate a few hundred dollars of inference spend — which is exactly why vendors get away with flattering numbers, because almost nobody pays to check.