Model update checks

Compare two model versions under a fixed evaluation budget.

A paired regression test asks whether a candidate has made things worse on the same items. EvalDelta keeps exploratory selection separate from the samples used to confirm a result.

Saved Python replaysSynthetic replays + real benchmark summary5,000 items · seed 42

Explore a saved run

Choose a scenario and a maximum number of candidate evaluations. The numbers below were generated by the EvalDelta Python engine before this page was built.

Scenario Maximum paid calls

Loading saved runs…

Real-data benchmark summary

Precomputed aggregate results from revision 2: Adult, Covertype, AG News, and Bank Marketing, plus a separate CIFAR-10 transfer test. No dataset rows or models run on this page.

Maximum paid calls

Loading aggregate results…

Each rate uses repeated seeds on fixed model-version pairs; those repeats are not independent datasets. The four-source interval spans zero, and one CIFAR-10 source cannot support a source-level interval. These results do not establish an efficiency advantage. Revision 2 reused three test source pairs after revision 1, so it is not a pristine second held-out test. See the results and limitations.

How to read the synthetic replay: Positive paired loss change means the candidate is worse. The full-pool value shown above is an after-the-run audit reference; selection policies do not receive hidden candidate outcomes. The interval shown on each card is for the global effect; a regression decision can come from a confirmed slice even when that interval spans zero. Read decisions under the stated sampling assumptions. An inconclusive result is never approval to deploy.