Benchmark Contracts and Vendor Acceptance Testing

Buying or reserving quantum capacity is not like buying a classical CPU cluster. Qubit count without fidelity, topology, calibration stability, queue policy, tooling, and data rights is not capacity.

Vendor evaluation must be evidence-based. Benchmarks should be written as contracts, not as demos.

40.1 Evaluation funnel

Diagram loads as you read

40.1 Evaluation funnel · Figure 1

View diagram source

flowchart TD
    Market[Vendor market scan] --> Shortlist[Technical shortlist]
    Shortlist --> NDA[NDA and data access]
    NDA --> Bench[Benchmark campaign]
    Bench --> Ops[Operations review]
    Ops --> Security[Security review]
    Security --> Commercial[Commercial negotiation]
    Commercial --> Pilot[Pilot]
    Pilot --> Decision[Scale, hold, or exit]

Do not run benchmarks before defining the decision they support. Benchmarks without acceptance criteria become theater.

40.2 Benchmark contract structure

Illustrative listing · yaml

benchmark_contract:
  benchmark_id: string
  decision_supported: renew | buy | reserve | publish | exit
  workload_classes:
    - circuit_sampling
    - chemistry_estimation
    - optimization_sampling
  acceptance_criteria:
    fidelity_threshold: float
    queue_wait_p95_minutes: float
    reproducibility_score: float
    cost_per_successful_trial: number
  data_rights:
    raw_shots_access: true
    calibration_snapshot_access: true
    publication_allowed: bool
  execution_window:
    reservation_required: bool
    start: timestamp
    end: timestamp

A contract should define the workload, device conditions, allowed compiler settings, measurement method, data access, and pass/fail criteria.

40.3 Reservations change the benchmark meaning

Amazon Braket documents reservations as scheduled exclusive access to a chosen quantum device [R71]. The general principle applies across providers: dedicated windows remove some queue noise but can introduce selection bias if the reserved window is unusually well prepared.

Diagram loads as you read

40.3 Reservations change the benchmark meaning · Figure 2

View diagram source

flowchart LR
    OnDemand[On-demand queue] --> Measures[Measures normal user experience]
    Reservation[Reserved window] --> Measures2[Measures controlled capacity]
    Measures --> Decision[Procurement decision]
    Measures2 --> Decision

Benchmark both modes when possible:

Mode	Measures	Does not measure
on-demand	queue experience, ordinary operations	peak controlled performance
reservation	controlled execution, throughput	normal queue fairness
vendor-assisted	best-case support process	self-service usability
blind benchmark	realistic reproducibility	collaborative debugging

40.4 Score what matters

Diagram loads as you read

40.4 Score what matters · Figure 3

View diagram source

flowchart TB
    Vendor[Vendor evaluation] --> Hardware[Hardware quality]
    Vendor --> Tooling[Tooling quality]
    Vendor --> Ops[Operational maturity]
    Vendor --> Data[Data and provenance]
    Vendor --> Security[Security and governance]
    Vendor --> Economics[Economics]
    Vendor --> Roadmap[Roadmap credibility]

Dimension	Example evidence
hardware quality	fidelity, topology, calibration stability, uptime
tooling quality	compiler control, APIs, SDK maturity, reproducibility
operations	incident process, maintenance windows, calibration reports
data	raw shots, metadata, retention, exportability
security	access controls, tenant isolation, compliance posture
economics	cost per accepted result, reservation policy, support cost
roadmap	delivered milestones, not only announced ambitions

40.5 Use benchmark families, not one magic score

Diagram loads as you read

40.5 Use benchmark families, not one magic score · Figure 4

View diagram source

flowchart LR
    BenchmarkSuite[Benchmark suite] --> Micro[Microbenchmarks]
    BenchmarkSuite --> Algorithmic[Algorithmic benchmarks]
    BenchmarkSuite --> Operational[Operational benchmarks]
    BenchmarkSuite --> Economic[Economic benchmarks]
    Micro --> Report[Benchmark report]
    Algorithmic --> Report
    Operational --> Report
    Economic --> Report

Benchmark families:

Family	Examples
micro	gate fidelity, readout, crosstalk, connectivity
circuit	depth survival, routing overhead, dynamic-circuit behavior
algorithmic	VQE/QAOA/sampling task with baseline
operational	queue wait, failure rate, metadata completeness
economic	cost per successful validated result

Diagram loads as you read

40.6 Blind and reproducible campaigns · Figure 5

View diagram source

sequenceDiagram
    participant Buyer as Buyer team
    participant Vendor as Vendor
    participant Arbiter as Independent reviewer
    Buyer->>Arbiter: benchmark spec and scoring rubric
    Arbiter->>Vendor: workload package
    Vendor->>Arbiter: raw results and metadata
    Arbiter->>Buyer: scored report
    Buyer->>Vendor: remediation questions

Blind campaigns reduce tuning to the benchmark. Reproducible campaigns reduce ambiguity. For high-stakes procurement, use both.

40.7 Data-rights clauses

Benchmark data is valuable. Contract language should specify who can store, analyze, publish, and reuse it.

Diagram loads as you read

40.7 Data-rights clauses · Figure 6

View diagram source

flowchart TB
    Data[Benchmark data] --> Raw[Raw shots]
    Data --> Meta[Metadata]
    Data --> Cal[Calibration snapshots]
    Data --> Logs[Execution logs]
    Data --> Derived[Derived metrics]
    Raw --> Rights[Data-rights policy]
    Meta --> Rights
    Cal --> Rights
    Logs --> Rights
    Derived --> Rights

Required clauses:

raw-shot access,
calibration metadata access,
compiler/transpiler version disclosure,
anonymization policy,
publication review process,
retention and deletion policy,
right to rerun after vendor remediation.

40.8 Acceptance and exit

Diagram loads as you read

40.8 Acceptance and exit · Figure 7

View diagram source

flowchart TD
    Results[Benchmark results] --> Pass{Pass acceptance?}
    Pass -- yes --> Pilot[Enter pilot]
    Pass -- no --> Remediate{Remediation credible?}
    Remediate -- yes --> Rerun[Rerun limited benchmark]
    Remediate -- no --> Exit[Exit process]
    Rerun --> Pass

Exit should be defined before the test begins. Without an exit path, benchmark programs drift into endless vendor management.

Acceptance packet:

executive decision summary,
benchmark specification,
raw and processed results,
scoring workbook,
unresolved risks,
remediation plan,
commercial implications,
go/no-go recommendation.