Quantum Benchmark Marketplaces and Public Scoreboards

Benchmarks become more useful when they are portable, reproducible, and publicly comparable. They become more dangerous when they collapse different workloads, targets, and operating conditions into a single league table.

A mature quantum ecosystem needs benchmark marketplaces: places where benchmark definitions, inputs, execution evidence, provider results, cost, runtime, and uncertainty can be compared without erasing context. QED-C has published application-oriented benchmark suites that evaluate algorithms across problem sizes and measure quality, execution time, and resource consumption. IBM and community efforts have also promoted benchmark libraries for optimization workloads. [R150]

Diagram loads as you read

Quantum Benchmark Marketplaces and Public Scoreboards · Figure 1

View diagram source

flowchart LR
    Definition[Benchmark definition] --> Runner[Portable runner]
    Runner --> Providers[Provider targets]
    Providers --> Results[Results]
    Results --> Evidence[Evidence package]
    Evidence --> Scoreboard[Scoreboard]
    Scoreboard --> Review[Review and challenge]

Benchmark marketplace components

Diagram loads as you read

Benchmark marketplace components · Figure 2

View diagram source

flowchart TB
    Market[Benchmark marketplace] --> Catalog[Benchmark catalog]
    Market --> Runner[Execution runners]
    Market --> Evidence[Evidence repository]
    Market --> Scoreboard[Scoreboards]
    Market --> Governance[Governance]
    Market --> Challenges[Challenge process]

A benchmark marketplace should store:

benchmark definitions;
versioned inputs;
implementation mappings;
target descriptors;
compiler settings;
mitigation settings;
shot budgets;
classical post-processing code;
cost and runtime;
uncertainty intervals;
raw evidence links.

Scorecards, not leaderboards

Diagram loads as you read

Scorecards, not leaderboards · Figure 3

View diagram source

flowchart TB
    Scorecard[Benchmark scorecard] --> Correctness[Correctness]
    Scorecard --> Runtime[Runtime]
    Scorecard --> Cost[Cost]
    Scorecard --> Repeatability[Repeatability]
    Scorecard --> Portability[Portability]
    Scorecard --> Evidence[Evidence quality]

A leaderboard ranks. A scorecard explains.

Dimension	What it prevents
correctness	fast wrong answers
runtime	unusable demonstration wins
cost	uneconomic scaling claims
repeatability	one-off lucky results
portability	vendor-specific overfitting
evidence quality	unverifiable claims

Benchmark lifecycle

Diagram loads as you read

Benchmark lifecycle · Figure 4

View diagram source

stateDiagram-v2
    [*] --> Proposed
    Proposed --> Drafted
    Drafted --> Piloted
    Piloted --> Accepted
    Accepted --> Deprecated
    Accepted --> Revised
    Revised --> Piloted
    Deprecated --> Archived

Benchmarks age. Algorithms improve, hardware constraints shift, and classical baselines move. A marketplace must support deprecation and revision.

Submission flow

Diagram loads as you read

Submission flow · Figure 5

View diagram source

sequenceDiagram
    participant Submitter
    participant Registry
    participant Validator
    participant Runner
    participant Review
    participant Board

    Submitter->>Registry: submit benchmark package
    Registry->>Validator: schema and reproducibility checks
    Validator->>Runner: run reference execution
    Runner-->>Validator: result bundle
    Validator->>Review: evidence package
    Review->>Board: approve, reject, or request changes
    Board-->>Registry: publish version

Anti-gaming controls

Diagram loads as you read

Anti-gaming controls · Figure 6

View diagram source

flowchart TB
    Gaming[Benchmark gaming] --> Overfit[Input overfitting]
    Gaming --> HiddenTuning[Hidden tuning]
    Gaming --> CherryPick[Cherry-picked target window]
    Gaming --> BaselineWeak[Weak classical baseline]
    Gaming --> CostHide[Hidden cost]

    Overfit --> Counter1[Private holdout inputs]
    HiddenTuning --> Counter2[Config disclosure]
    CherryPick --> Counter3[Time-window disclosure]
    BaselineWeak --> Counter4[Baseline review]
    CostHide --> Counter5[Cost accounting]

Public result schema

This illustrative record uses a Max-Cut instance with a known, positive optimum. Its approximation ratio is the expected cut value divided by that optimum. The interval describes uncertainty in that mean ratio under the declared sampling method. These example values are not measured QED-C results. Energy-estimation workloads such as VQE need a separately defined energy-error metric and reference.

Illustrative listing · yaml

benchmark_result:
  benchmark_id: illustrative_maxcut
  benchmark_version: 3.1.0
  target: provider.qpu.family.device
  execution_window: 2026-04-19T10:00:00Z/2026-04-19T12:00:00Z
  compiler:
    name: qiskit
    version: 2.x
  mitigation:
    profile: zne_measurement_v2
  shots: 20000
  quality_metric:
    name: approximation_ratio
    value: 0.91
    confidence_interval: [0.88, 0.94]
  runtime_seconds: 1240
  cost_usd_estimate: 183.20
  evidence_uri: evidence://benchmark/illustrative_maxcut/run_001

Diagram loads as you read

Public result schema · Figure 7

View diagram source

flowchart LR
    Result[Result] --> Metric[Quality metric]
    Result --> Runtime[Runtime]
    Result --> Cost[Cost]
    Result --> Evidence[Evidence URI]
    Result --> Context[Execution context]

Cross-platform comparison

Diagram loads as you read

Cross-platform comparison · Figure 8

View diagram source

flowchart TB
    Workload[Benchmark workload] --> TargetA[Target A]
    Workload --> TargetB[Target B]
    Workload --> TargetC[Target C]
    TargetA --> Normalize[Normalize context]
    TargetB --> Normalize
    TargetC --> Normalize
    Normalize --> Compare[Compare with caveats]

Do not compare raw circuit fidelity across modalities without explaining compilation, native gates, topology, and execution model.

Challenge process

Diagram loads as you read

Challenge process · Figure 9

View diagram source

flowchart LR
    Published[Published result] --> Challenge[Challenge filed]
    Challenge --> Reproduce[Independent reproduction]
    Reproduce --> Decision{Sustained?}
    Decision -- yes --> Correct[Correct or retract]
    Decision -- no --> Annotate[Annotate challenge]

A healthy marketplace allows correction without stigma. A benchmark result is a scientific artifact, not a press release.

Operating rule

A benchmark marketplace should reward reproducible, contextual evidence. Any scoreboard that hides uncertainty, cost, or execution conditions is not an engineering tool.