Benchmarking Beyond Gate Fidelity — Operating Quantum Computers

Gate fidelity matters, but it is not a product metric. A user does not buy a single two-qubit gate. A user runs a workload, waits in a queue, pays for shots, applies mitigation, and asks whether the resulting decision is credible.

The QED-C application-oriented benchmark suite was designed to evaluate quantum computers on application-like programs across problem sizes, including metrics such as solution quality, execution time, and gate resources [R135]. That is the right direction: benchmarks must connect device behavior to workload value.

79.1 Benchmark stack

Diagram loads as you read

79.1 Benchmark stack · Figure 1

View diagram source

flowchart TB
    Benchmarks[Benchmark portfolio] --> Device[Device benchmarks]
    Benchmarks --> Circuit[Circuit benchmarks]
    Benchmarks --> Application[Application benchmarks]
    Benchmarks --> Workflow[Workflow benchmarks]
    Benchmarks --> Economic[Economic benchmarks]
    Benchmarks --> Evidence[Evidence benchmarks]

A single number is never enough. The benchmark stack should expose tradeoffs.

79.2 Benchmark dimensions

Diagram loads as you read

79.2 Benchmark dimensions · Figure 2

View diagram source

flowchart LR
    Benchmark[Benchmark] --> Quality[Result quality]
    Benchmark --> Time[Time to result]
    Benchmark --> Cost[Cost]
    Benchmark --> Stability[Stability]
    Benchmark --> Portability[Portability]
    Benchmark --> Evidence[Evidence completeness]

Every benchmark should specify which dimensions it measures and which it deliberately ignores.

79.3 Application-centered benchmark contract

Diagram loads as you read

79.3 Application-centered benchmark contract · Figure 3

View diagram source

flowchart TB
    Contract[Benchmark contract] --> Intent[Application intent]
    Contract --> Dataset[Input family]
    Contract --> Metric[Quality metric]
    Contract --> Budget[Cost and time budget]
    Contract --> Targets[Allowed targets]
    Contract --> Baseline[Classical baseline]
    Contract --> Evidence[Required evidence]

The contract prevents benchmark theater: changing inputs, targets, or stopping rules until a desired result appears.

79.4 Benchmark lifecycle

Diagram loads as you read

79.4 Benchmark lifecycle · Figure 4

View diagram source

stateDiagram-v2
    [*] --> Proposed
    Proposed --> Validated: metric and baseline accepted
    Validated --> Scheduled: cadence set
    Scheduled --> Running: execution window
    Running --> Published: evidence reviewed
    Published --> Deprecated: obsolete target or workload
    Running --> Quarantined: anomaly detected

A benchmark is a product. It needs ownership, cadence, review, and retirement.

79.5 Workload families

Diagram loads as you read

79.5 Workload families · Figure 5

View diagram source

flowchart TB
    Families[Benchmark families] --> Sampling[Sampling]
    Families --> Optimization[Optimization]
    Families --> Chemistry[Chemistry]
    Families --> Dynamics[Quantum dynamics]
    Families --> QML[Quantum ML]
    Families --> ErrorCorrection[Error-correction kernels]
    Families --> Control[Control and feedback]

The portfolio should match strategy. A chemistry program should not optimize its platform using only random-circuit benchmarks.

79.6 Benchmark evidence pipeline

Diagram loads as you read

79.6 Benchmark evidence pipeline · Figure 6

View diagram source

sequenceDiagram
    participant Scheduler as Benchmark scheduler
    participant Runtime as Runtime
    participant Store as Evidence store
    participant Review as Review board
    participant Dashboard as Benchmark dashboard
    Scheduler->>Runtime: execute benchmark contract
    Runtime->>Store: write raw and processed artifacts
    Store->>Review: request review
    Review-->>Store: approve or quarantine
    Store->>Dashboard: publish reviewed metrics

Dashboards should show uncertainty and comparability warnings. A benchmark point without target context is a trap.

79.7 Leading and lagging indicators

Diagram loads as you read

79.7 Leading and lagging indicators · Figure 7

View diagram source

flowchart LR
    Leading[Leading indicators] --> Drift[Drift alarms]
    Leading --> Calibration[Calibration regressions]
    Leading --> Queue[Queue delay]
    Leading --> Cost[Cost per evidence]
    Lagging[Lagging indicators] --> Quality[Application quality]
    Lagging --> Claim[Claim acceptance]
    Lagging --> Revenue[Product impact]

Leading indicators help operators act before application metrics degrade. Lagging indicators tell leaders whether the platform is worth funding.

79.8 Operating rule

A benchmark is useful only if it changes a decision.

Diagram loads as you read

79.8 Operating rule · Figure 8

View diagram source

flowchart LR
    Benchmark[Benchmark result] --> Decision{Decision affected?}
    Decision -- yes --> Keep[Keep and fund]
    Decision -- no --> Retire[Retire or redesign]