A benchmark connects a measurement to a decision. Its usefulness depends on whether the measured workload, comparison method, resource accounting, and acceptance criterion match the decision someone will make. An impressive number without that connection can be perfectly reproducible and still answer the wrong question.
Start by writing a claim that could be false. “This implementation meets a measured Bell-score target under this simulator model and stopping rule” is testable. “The platform is quantum-ready” leaves the workload and standard of success undefined. The lab uses the narrower claim: the observed score must clear 0.90 with an approved uncertainty bound, within a specified shot budget, and with valid evidence.
Preserve the path from workload to claim
A comparison needs the problem instance, input encoding, circuit or algorithm, target configuration, estimator, uncertainty rule, and resource limits. Record compiler and simulator versions because the executable program and sampling stream depend on them. Include unsuccessful attempts and declared exclusions. A dashboard containing only accepted runs cannot reveal how much computation was spent reaching them.
Distinguish correctness checks from performance comparisons. The Bell density-matrix calculation provides a tractable reference for the chosen noise model. It does not benchmark a useful quantum advantage. A larger application comparison needs a relevant classical method, meaningful solution quality, and equivalent accounting for preparation, training, compilation, and postprocessing. The relevant baseline is the method a user could reasonably choose for the same task.
Identical random seeds across different engines do not require identical samples. Compare analytical distributions and statistically appropriate summaries. Exact replay within one pinned engine is valuable for debugging, but it is not independent replication.
Read what the captured evidence actually establishes
The committed Aer baseline capture ends at 5,000 shots per basis with score 0.956100 and interval [0.917177, 0.995023]. It supports acceptance under the lab's raw-score rule. It does not establish pre-readout fidelity, hardware performance, or a probability that every future run accepts.
The captured invalid-provenance scenario has the same statistical history but finishes with disposition invalid because its model identifier is deliberately removed. This is a constructed provenance fault applied to real simulator output. The estimator still computes a number; the evidence no longer meets the reporting contract. Execution status, statistical decision, and evidence validity must remain separate fields.
The coverage fixture is another kind of evidence: a Monte Carlo model check using an independent calculation of the interval rule. At true correlation magnitude zero it records 52 simultaneous-correlation misses in 10,000 trials; at magnitude 0.5 it records four. These are not Aer workload comparisons. Nor does observing no score misses prove that the score-bound failure probability is zero. The mathematical argument and the numerical implementation check serve different purposes.
Compare a difference, not two impressions
Suppose the useful question is whether two methods differ by more than a practical tolerance. Define Δ=S_B−S_A and choose the tolerance before evaluating. A confidence interval for Δ wholly inside [−τ,τ] supports equivalence within that tolerance under the design. A wide interval containing zero does not.
The following is an analytic, constructed comparison from the teaching fixture, with estimates S_A=0.955, S_B=0.950 and tolerance τ=0.020. Each row represents its own predeclared fixed-sample design, not repeated unadjusted looks at one experiment.
| Shots per basis per method | Estimated difference | Conservative 95% difference interval | Decision |
|---|---|---|---|
| 10,000 | −0.005 | [−0.054662, 0.044662] | inconclusive |
| 100,000 | −0.005 | [−0.020704, 0.010704] | inconclusive |
| 200,000 | −0.005 | [−0.016105, 0.006105] | within tolerance |
These bounds allocate error probability across all six correlations. They are deliberately conservative. Using two adaptively stopped seven-look runs would require protecting the larger family of decisions. Two separate 95% intervals do not automatically yield a 95% comparison, and interval overlap is not the comparison rule.
The final row costs 1.2 million shots across both methods and all bases. A tight practical claim can require substantially more evidence than merely demonstrating that one configuration clears a broad threshold. This cost belongs in the benchmark plan, before acquisition.
Exercise and worked answer
A report says: “Method B is equivalent to A because the difference estimate is only −0.005 and its interval includes zero.” It used the 100,000-shot row above. Correct the conclusion, then state what the 200,000-shot row would permit. Finally, explain whether the captured baseline score proves hardware superiority.
Worked answer: At 100,000, the interval extends to −0.020704, beyond the declared −0.020 tolerance. The comparison is inconclusive about equivalence; lack of a demonstrated difference is not evidence of a sufficiently small difference. At 200,000, the interval lies entirely within the tolerance band, supporting that narrowly defined equivalence claim under the fixed design and assumptions. The Aer baseline establishes neither hardware execution nor a comparison against another useful method. Its defensible statement names the simulator, workload, readout-affected score, shot count, interval, and decision rule. Capacity, Scheduling, and Cost connects the size of that evidence requirement to resource planning.