Incidents, Recovery, and Safe Changes

An incident begins with an observation and a decision about what work can still be trusted. A low score, an invalid record, and an interrupted worker are different observations. Treating them as interchangeable failures makes recovery less reliable because it obscures which part of the experiment needs attention.

The lab supplies these cases as executable scenarios. They let an operator practice preserving useful evidence, identifying the failed condition, and choosing a continuation that does not quietly change the scientific question. The earlier chapters define the record and statistical rule; this chapter applies them during a disruption.

Diagnose the condition before choosing a retry

Run the three distinct failure examples locally:

quantum-ops --db incidents.sqlite3 run --scenario degraded
quantum-ops --db incidents.sqlite3 run --scenario invalid
quantum-ops --db incidents.sqlite3 run --scenario interrupted

In the captured degraded run, the observed score is 0.6125 at 200 shots per basis. Its upper bound is approximately 0.807116, below the 0.9 threshold. The evidence is valid and the quality claim is rejected. Continuing to collect shots after this terminal decision would violate the declared procedure.

The invalid run instead loses its model identifier. Repeating the score calculation cannot establish that missing provenance. The interrupted run preserves completed batches and can resume under the same compatible configuration and engine. These differences determine whether the next action is investigation, reconstruction of trustworthy context, or continuation of existing work.

Worked example: separate a changed regime from recovery

The drift scenario runs the degraded readout regime, then starts a separate baseline recovery after completed execution:

quantum-ops --db incidents.sqlite3 run --scenario drift

The command returns records for the original run and its recovery. The recovery has a new identifier, a new seed, its own batches, and the context recovery-after-drift. Its recoveryOf field identifies the earlier run. This establishes a relationship without merging their measurements.

The scenario samples a stable degraded regime from its first shot. It demonstrates recovery after an observed change; it does not implement a detector that discovers when a parameter changes within one run. The separate synthetic monitoring trace in Detecting Drift and Interpreting Alerts teaches that detection problem, including false alarms.

Pooling degraded and recovered observations would answer a historical mixture question. It would not estimate the recovered state alone. Replacing the old configuration while keeping its counts would be worse: the metadata would describe measurements that were never performed under that configuration. Preserve both records so a reviewer can see the quality before and after the change.

Choose changes that can be evaluated

A useful recovery action has a stated effect and a verification step. For this simulation, restoring the baseline model is explicit and its new measurements are independently recorded. For physical hardware, a calibration change would need a versioned context and fresh characterization appropriate to the claim. More shots do not repair systematic measurement bias or recreate missing provenance.

Rollback also has limits. Reverting a classical configuration can restore future behavior; it cannot undo a completed destructive measurement or reconstruct an unknown quantum state. The local resume path works because the retained objects are classical counts, configuration, and execution history.

Cancellation is another operational decision. The runner checks for it around complete sampling work and retains committed evidence. A cancellation request is not proof that no additional work occurred before the worker observed it. The final status must reflect what the service actually recorded.

Exercise and worked answer

An operator receives a degraded record with a terminal rejection. They propose changing its context to the recovery label, increasing its budget, and appending baseline shots until the combined score passes. Which parts of that plan fail?

Worked answer: The existing run has already stopped, so additional batches violate the stopping history. Changing its context misrepresents the earlier samples. Increasing its budget after inspecting the result changes the approved experiment. Finally, the combined statistic targets a mixture of operating regimes rather than the recovered regime.

Start a new run with an explicit recovery relationship and freshly declared settings. Leave the rejected record intact. If the new record passes its own criteria, report recovery for that new experiment under its assumptions. Record the cause of the intervention separately from the measured result; a favorable score alone does not prove the diagnosis of the earlier incident.