Vendor Evaluation, Procurement, and SLAs — Operating Quantum Computers

Quantum procurement is not normal cloud procurement. A buyer is not merely purchasing compute cycles. The buyer is purchasing access to a physical system whose quality changes over time, whose queue may be shared, whose hardware may have target-specific constraints, and whose evidence value depends on calibration, metadata, and operational transparency.

This chapter defines a procurement and vendor-evaluation process for quantum platforms. It applies to cloud access, dedicated reservations, on-premises systems, integration partners, software tooling, and managed research engagements.

32.1 Evaluation lifecycle

Diagram loads as you read

32.1 Evaluation lifecycle · Figure 1

View diagram source

flowchart LR
    Need[Use-case need] --> RFI[RFI / landscape scan]
    RFI --> Shortlist[Shortlist]
    Shortlist --> Trial[Technical trial]
    Trial --> Benchmark[Benchmark campaign]
    Benchmark --> Risk[Risk and security review]
    Risk --> Contract[Contract / SLA]
    Contract --> Operate[Operate]
    Operate --> Review[Quarterly vendor review]

Do not skip the benchmark campaign. Marketing metrics rarely answer operational questions.

32.2 RFI structure

A quantum RFI should ask for measurable operational capabilities.

Diagram loads as you read

32.2 RFI structure · Figure 2

View diagram source

flowchart TB
    RFI[RFI] --> Hardware[Hardware capability]
    RFI --> Access[Access model]
    RFI --> Metadata[Metadata and provenance]
    RFI --> Security[Security and governance]
    RFI --> Support[Support model]
    RFI --> Roadmap[Roadmap and deprecation policy]

RFI sections:

Section	Questions
hardware	modality, qubits, connectivity, native operations, benchmarks
access	cloud, reservation, dedicated, on-prem, queue model
limits	shots, circuits, payload, runtime, sessions, reservations
metadata	calibration, backend properties, result schemas, raw artifacts
security	IAM, isolation, audit logs, data handling
support	incident process, escalation, maintenance windows
roadmap	API stability, hardware upgrades, deprecation notices
exit	data export, artifact ownership, portability support

32.3 Acceptance tests

Acceptance tests convert vendor claims into observed behavior.

Diagram loads as you read

32.3 Acceptance tests · Figure 3

View diagram source

flowchart LR
    Claim[Vendor claim] --> Test[Test design]
    Test --> Run[Run on target]
    Run --> Evidence[Evidence bundle]
    Evidence --> Accept{Accept?}
    Accept -- yes --> Contract[Contract baseline]
    Accept -- no --> Gap[Gap remediation]

Acceptance categories:

Category	Example test
access	create job, batch, session, or reservation
metadata	retrieve target properties and attach to run
result fidelity	run standard circuits and compare expected distributions
queue behavior	measure wait-time distribution under trial conditions
support	exercise escalation path with a non-destructive issue
data export	export raw counts and metadata
security	verify IAM boundaries and audit trails

32.4 Benchmark campaign design

Vendor evaluation benchmarks should match your intended workloads, not only standard public benchmarks.

Diagram loads as you read

32.4 Benchmark campaign design · Figure 4

View diagram source

flowchart TB
    Benchmarks[Benchmark campaign] --> Standard[Standard circuits]
    Benchmarks --> Workload[Representative workloads]
    Benchmarks --> Stress[Operational stress tests]
    Benchmarks --> Drift[Drift and repeatability tests]
    Benchmarks --> Cost[Cost and throughput tests]

Benchmark campaign artifact:

Illustrative listing · yaml

vendor_benchmark_campaign:
  vendor: string
  targets:
    - string
  period:
    start: date
    end: date
  benchmark_suites:
    - name: standard_circuits
      purpose: sanity_check
    - name: representative_workloads
      purpose: product_fit
    - name: repeatability
      purpose: drift_assessment
  acceptance_thresholds:
    queue_p95_seconds: int|null
    result_quality_minimum: string
    metadata_completeness: float
  evidence_uri: string

32.5 SLA dimensions

Quantum SLAs need more than uptime. Uptime without useful device quality is not enough.

Diagram loads as you read

32.5 SLA dimensions · Figure 5

View diagram source

mindmap
  root((Quantum SLA))
    Availability
      Service uptime
      Target availability
      Maintenance notice
    Quality
      Calibration freshness
      Benchmark publication
      Drift notification
    Throughput
      Queue behavior
      Reservation access
      Batch limits
    Metadata
      Result completeness
      Calibration snapshots
      Audit logs
    Support
      Escalation
      Incident communication
      RCA

Potential SLA terms:

Dimension	Example measure
platform availability	API and portal uptime
target availability	scheduled access windows or target uptime
metadata completeness	required fields present in result bundles
calibration freshness	maximum age or disclosure policy
maintenance notice	lead time for planned outages
queue transparency	status visibility and historical wait data
support response	response and escalation targets
deprecation notice	API and target retirement policy

32.6 Quality is not a single number

A vendor may advertise high-level performance metrics, but your workload may depend on a specific subgraph, gate type, measurement behavior, or queue mode.

Diagram loads as you read

32.6 Quality is not a single number · Figure 6

View diagram source

flowchart TB
    Quality[Quality] --> Global[Global benchmark]
    Quality --> Local[Local qubit subset]
    Quality --> Temporal[Temporal stability]
    Quality --> Workload[Workload-specific success]
    Quality --> Metadata[Transparency]

Ask for quality evidence at the level your workload uses.

32.7 Data rights and artifact ownership

Contracts should specify who owns experiment inputs, compiled circuits, result data, metadata, logs, derived reports, and benchmark artifacts.

Diagram loads as you read

32.7 Data rights and artifact ownership · Figure 7

View diagram source

erDiagram
    CUSTOMER ||--o{ INPUT_ARTIFACT : owns
    CUSTOMER ||--o{ RESULT_ARTIFACT : owns
    PROVIDER ||--o{ HARDWARE_METADATA : supplies
    PROVIDER ||--o{ SERVICE_LOG : controls
    CONTRACT ||--o{ DATA_RIGHT : defines
    DATA_RIGHT ||--o{ ARTIFACT_CLASS : applies_to

Contract questions:

Question	Reason
Can raw result data be exported?	reproducibility and audit
Can calibration metadata be retained?	evidence integrity
Can provider use workloads for benchmarking?	confidentiality
Are compiled circuits confidential?	IP protection
What happens at contract end?	exit planning

32.8 Vendor lock-in controls

Some lock-in is acceptable when capability is unique. Hidden lock-in is not.

Diagram loads as you read

32.8 Vendor lock-in controls · Figure 8

View diagram source

flowchart LR
    ProviderFeature[Provider-specific feature] --> Dependency[Recorded dependency]
    Dependency --> Portability[Portability assessment]
    Portability --> Exit[Exit or fallback plan]

Lock-in register:

Illustrative listing · yaml

vendor_dependency:
  dependency_id: string
  vendor: string
  feature: string
  workloads_affected:
    - string
  portability_level: L0|L1|L2|L3|L4|L5
  fallback: string|null
  business_justification: string
  review_date: date

32.9 Quarterly vendor review

Quantum vendors and hardware change quickly. Procurement should not stop at signature.

Diagram loads as you read

32.9 Quarterly vendor review · Figure 9

View diagram source

flowchart TB
    Metrics[Operational metrics] --> QBR[Quarterly business review]
    Incidents[Incidents] --> QBR
    Roadmap[Roadmap changes] --> QBR
    Benchmarks[Benchmark trends] --> QBR
    Spend[Spend and utilization] --> QBR
    QBR --> Decisions[Renew, expand, remediate, or exit]

Review agenda:

utilization and spend,
queue and reservation performance,
benchmark trends,
incidents and RCAs,
metadata completeness,
roadmap changes,
deprecations,
security updates,
support quality,
next-quarter workload plan.

32.10 Chapter checklist

A quantum procurement package should include:

RFI with operational questions,
representative benchmark campaign,
acceptance tests,
security and data-handling review,
SLA dimensions beyond uptime,
quality evidence requirements,
artifact ownership terms,
lock-in register,
quarterly review cadence,
exit plan.

Diagram loads as you read

32.10 Chapter checklist · Figure 10

View diagram source

flowchart LR
    RFI[RFI] --> Package[Procurement package]
    Benchmark[Benchmark] --> Package
    SLA[SLA] --> Package
    Security[Security] --> Package
    Exit[Exit plan] --> Package