Tracker model benchmark

Two tracker models, one identical fixed-seed synthetic scene with perfect ground truth. The sensor model, detection generation, difficulty settings, and metric definitions are byte-identical between models — only the tracker differs. Reproduce every row yourself: open the live demonstration and press "Run benchmark". No signup, no NDA, no clearance.

SEED 1337 · 20 s WARM-UP + 120 s MEASURED PER ROW · DETECTION IS A SENSOR PROPERTY — IDENTICAL FOR BOTH MODELS BY CONSTRUCTION · V2 RUNTIME ~1.2 ms AVG PER SENSOR TICK AT DENSITY 400 (BUDGET 10 ms)

ID switches/min counts every change of the track identity assigned to a ground-truth object — deliberately stricter than the MOT-challenge IDSW definition, because fragmentations and re-acquisitions count as switches too. Detection is detections ÷ true objects per sensor frame. The fixed seed makes every row reproducible: same numbers on every run in the same browser, and the matrix above was additionally reproduced headless (two identical runs) before publication.

Why we publish failure numbers: both models still commit thousands of identity errors per minute under stress, and those rows stay on this page on purpose. Synthetic scenes provide perfect ground truth, so these are measurements, not marketing. Every future tracker model must land here as a new column against the same seed — in public. Every demonstration slice stays runnable as shipped: slice 1, slice 2 (tracker v1), slice 3 (tracker v2, current).

The open tracker-v3 packet asks for −30 % identity switches on the three hard rows (frame-starved ≤ 2 023, occluded ≤ 1 537, degraded ≤ 5 146) with no regression on baseline/dense. We handed the identical packet — same v2 source, same rules, one repair round — to every code-capable chat model on NVIDIA Build (free tier): 53 models across the catalog, plus our own executor lane, and benchmarked every returned tracker on the same seed-1337 matrix. Integrity gates: bit-reproducible runs, detection column byte-identical to v2 (the sensor is untouchable), zero RNG consumption, no ground-truth access, and association coverage (a tracker must actually associate detections, not skip them). No executor has met the v3 targets; v2 remains the shipped tracker.

ID SWITCHES/MIN · SEED 1337 · SAME HARNESS AND CONFIGS AS SECTION 02 · NVIDIA BUILD FREE TIER, 2026-07-21→23 · 53 CATALOG MODELS + CODEX LANE · "EFFORT" = REASONING-EFFORT PARAMETER WHERE THE API EXPOSES ONE, OTHERWISE PROVIDER DEFAULT · ONE REPAIR ROUND PER MODEL

Three executors gamed the metric — all caught. First, gpt-oss-120b (low effort) posted an impossible 26 switches/min at baseline: its tracker wrote the ground-truth bookkeeping table (truthLast[d.truth]=d.trk) from inside the tracker, zeroing the counter instead of tracking anything. Ground-truth access became an instant-disqualification rule and the fleet was re-run. Then two models — nemotron-3-super-120b and llama-3.1-8b — independently found the same subtler hole: post an identical 26 / 19 / 152 / 35 / 82 by associating only 2.1 % of detections (~620 phantom tracks for 150 movers). An unassociated detection keeps its ground-truth object's last-track entry at null forever, and null = null never counts as a switch — a tracker that refuses to track scores perfectly. The harness now gates on association coverage and caught the second occurrence automatically. Every scored row passed audit: no ground-truth reads, no RNG, detection byte-identical to v2, association coverage in the normal band (98–99 %).

The full story — how every one of the 53 free NVIDIA models fared, which cheated and how, and which models were never suitable for the task — is written up as a dispatch: 53 free AI models, one tracker →

FULLY SYNTHETIC — EVERY PIXEL GENERATED. NO REAL PERSONS, VEHICLES, OR PLACES.