Blog

Gateway latency, measured: what the extra hop really costs

The standing objection to any router is the hop. We measured it instead of arguing about it: +68ms at the median, and the tail actually favored the gateway.

By the Unbiased Team · published August 9, 2026 · fees and prices verified July 28, 2026; measured data dated in place

We measured the gateway hop at +68ms median time-to-first-token (0.656s through a full routing gateway vs 0.588s direct), with the gateway's p95 beating direct by 104ms (0.821s vs 0.925s) across 20 streamed runs each way. Provider-side variance is larger than the hop; median latency is not a reason to avoid routers.

The method

Twenty streamed requests direct to a provider and twenty through a full routing gateway, identical prompt, same hour, measuring time-to-first-token. Small sample, deliberately simple, and reproducible in an afternoon; the runnable script is linked from the gateway comparison so you can check us against your own network and region.

We measured a full routing gateway rather than a pass-through proxy because that is the worst reasonable case: if routing logic runs on your request, you are paying for the whole detour.

The result, both halves

Median: the hop is real, 68 milliseconds, 0.656s against 0.588s. That is the honest cost of the convenience and for most interactive traffic it is imperceptible. Tail: the gateway's p95 came in 104ms better than direct, 0.821s against 0.925s, because provider-side variance, the thing you already live with, is bigger than the hop.

Read that second half carefully before architecting around latency: eliminating the gateway does not eliminate your tail. The provider is your tail.

Where latency actually becomes real

Per-request routing reasoning on hard tasks, not the network hop, is the latency question worth asking any routing layer. We document ours: Pareto can run up to 3x slower on the hardest reasoning tasks while measuring +0 on agentic work, stated on the model card rather than discovered in production. Demand the same disclosure from anything that thinks before it routes.

For simple catalog routing, the honest summary is: the hop is 68ms, your provider's p95 dwarfs it, and anyone using latency to sell you off a router owes you their measurement.

Questions, answered