Benchmarking AI gateway overhead under 50 ms

A governance gateway that adds noticeable latency will be bypassed. The 0.3 target was under 50 ms per call.

I added a test that builds the gateway the way kognita serve does and fails when overhead is not under 50 ms. The sample was 30 sequential calls, with classifier time excluded.

Recording the machine and the sample in the changelog mattered as much as the number. A benchmark without its context is marketing.

The overhead line of the 0.3 definition of done was ticked. Later CI runs showed stalled outliers, so the gate moved from the maximum to a 95th percentile and mean.

What I'd do differently

I'd gate on a percentile from the start. A maximum over 30 calls is at the mercy of one slow CI run.

Take this with you

Make latency a failing test, not a dashboard.

Sources

Kognita is an open-source alpha. This note comes from Kognita PR #26 in the Kognita repository.