- Cheap diagnostics
- Five always-on summary stats computed instantly for every run: how often orders used future-dated data, how many fills landed outside the trading session, the largest unexplained overnight price move, how many events are out of causal order, and how many fills are physically implausible. Each is bucketed none / mild / notable / severe.
- Jev's guess
- Jev never sees the raw run — only the five bucketed diagnostics. It picks the most likely failure family (or "clean") from a fixed list and returns a probability for each.
- Run diagnostic
- Executes the one deterministic check that can actually confirm Jev's guess — bounds checks, timestamp ordering, or a recorded-adjustment lookup. This is the only step that establishes a fact; Jev's guess is a triage prioritization, not a verdict.
- Accuracy
- Share of the 45 runs where Jev's top guess matches the family actually seeded into that run (or correctly says "clean").
- Missed / False alarms
- Missed = a real failure that Jev called clean. False alarm = a clean run that Jev flagged. Both are visible in "Run diagnostic" too — the deterministic check will not confirm a false alarm, and will still catch a missed failure if you spot-check a run Jev called clean.
- Brier score
- Average squared error between Jev's full probability vector and the true one-hot label across all six classes. Lower is better; a well-calibrated 6-class guess scores well below 0.5.