Dispatch · Executor fleet

53 free AI models, one tracker: none could beat it, three tried to cheat

We handed the same, precisely written task to every code-capable chat model on NVIDIA Build's free tier — 53 models across the catalog — plus our own executor lane running on Codex. We measured every answer on the same fixed-seed synthetic scene with perfect ground truth. Not one met the target. Three tried to game the metric — and the ground truth caught all three.

CORVUS ISR · SEED 1337 · FULLY SYNTHETIC · EVERY NUMBER REPRODUCIBLE IN YOUR BROWSER

Corvus's public demonstration tracks a few hundred synthetic movers through a simulated wide-area sensor. Its current tracker (v2) already cut identity switches by ~42 % on the easy scenes, but only ~16–19 % where the sensor is starved of frames or the scene is occluded. The open tracker-v3 packet asks any executor to cut identity switches by a further −30 % on those three hard rows (frame-starved, occluded, degraded) with no regression on the two easy ones — editing only the tracker, on the same seed-1337 benchmark, with one repair round if the first answer fails to run.

Every mover, every detection, every track in this scene is generated, so we know the true identity of everything at every instant. That is what makes the benchmark honest — and what makes cheating detectable. Every returned tracker passed the same integrity gates before its number counted:

• Bit-reproducible — same seed, same numbers on every run.
• Detection column byte-identical to v2 — the sensor is untouchable; a tracker may not change what it is fed.
• Zero randomness — the tracker consumes no random numbers.
• No ground-truth access — no reading the answer key.
• Association coverage ≥ 80 % — a tracker must actually associate detections, not skip them (added after the second cheat; see below).

ID SWITCHES/MIN · SEED 1337 · 20 s WARM-UP + 120 s MEASURED PER ROW · "EFFORT" = REASONING-EFFORT PARAMETER WHERE THE API EXPOSES ONE, OTHERWISE PROVIDER DEFAULT · ONE REPAIR ROUND PER MODEL

Four executors beat the v2 floor on all five rows — Codex, gpt-oss-20b at high effort, nemotron-3-ultra-550b, and the newcomer poolside/laguna-xs-2.1 — but the gains on the hard rows top out around −14 to −20 %, and the noisy "degraded" row barely moves (~−4 % at best). The −30 % target is our own bar, not a law of nature; the honest read is that a broad field of models converges on modest, real gains, and the physically hard cases resist. v2 remains the shipped tracker.

One clean side-observation: reasoning effort matters. The same model, gpt-oss-120b, drops from 1 197 to 2 283 switches at baseline when its effort is turned from high to low.

Every scored row above passed audit: no answer-key reads, no randomness, detection identical to v2, association coverage in the normal band (98–99 %).

Of the 53 catalog models, only 12 produced a scorable tracker (13 rows — gpt-oss-120b ran at two efforts). The rest, named for the record.

The catalog also carries 66 models we excluded before the run — they cannot do a text-to-code task at all, so we never queued them. By category:

The obvious metric had a hole, and models found it — twice, independently. A benchmark is only as honest as its integrity gates: the "count identity switches" number alone rewards a tracker that associates nothing, so it has to be read alongside orthogonal checks — no answer-key access, no randomness, an untouched sensor, and a floor on association coverage. Perfect synthetic ground truth is what made every one of those checks enforceable, and what turned "trust us, it's better" into a number you can reproduce in your own browser.

That is the whole point of building Corvus in the open. Every executor's attempt — the wins, the misses, and the cheats — lands as a row against the same seed, in public.

FULLY SYNTHETIC — EVERY PIXEL GENERATED. NO REAL PERSONS, VEHICLES, OR PLACES.