test(longhaul): add kill-operator-pod and kill-primary-pod chaos operations by WentingWu666666 · Pull Request #421 · documentdb/documentdb-kubernetes-operator

WentingWu666666 · 2026-07-10T16:40:20Z

Long-haul chaos operations + sequence-mode smoke gate

Adds the first two Chaos / HA operations from the long-haul design (docs/designs/long-haul-test-design.md → Operations) to the driver introduced in #405, reworks the outage-budget model they depend on, and adds a deterministic sequence operation mode plus a PR smoke gate that runs the real driver end-to-end against kind. Scope: fault-inject the control plane and the database primary on the canary cluster, and assert — over a multi-day run — that the data plane keeps serving and the cluster reconverges to its declared topology; and gate every PR by exercising each operation once against a real cluster.

Chaos operations

Two new operations registered with the scheduler:

Operation	Fault	Asserts
`kill-operator-pod`	Deletes the operator pod	The data plane keeps serving throughout (control-plane restart must not drop writes); the operator `Deployment` returns to `Available`.
`kill-primary-pod`	Deletes the CNPG primary pod	CNPG promotes a standby (automatic failover); writes resume and the cluster returns to full topology. Requires HA (`instancesPerNode>=2`).

The continuous workload verifier independently catches any data loss caused by a failover.

Outage policy: duration-based, writer-count-independent

The chaos ops needed a principled disruption budget, so OutagePolicy moved from a raw write-failure count to a wall-clock write-outage duration:

OutagePolicy.MaxWriteOutage time.Duration replaces AllowedWriteFailures int64. The journal converts the observed failure count into an estimated outage using the workload's aggregate write rate (WriteFailures / WritesPerSecond), so budgets no longer scale with LONGHAUL_NUM_WRITERS. The rate is computed once at startup from NumWriters and the fixed writer cadence (workload.AggregateWriteRate) and stamped onto each disruption window.
Two orthogonal guardrails, both fail independently (ExceededPolicy trips if either is exceeded):
- MaxWriteOutage — client-visible write availability (data plane).
- MustRecoverWithin — the managed cluster's return to full declared topology (all pods Ready, CR Ready). Because a failed op is only logged (not scored), this is the sole mechanism that fails the run when the cluster never reconverges — e.g. a failover where writes resume fast but a replacement standby never rejoins.
Shared budgets to prevent drift (all in journal/policy.go):
- NoOutagePolicy (~300ms cushion) for operations that keep the write path up throughout: kill-operator-pod and the scale ops (which only add/remove a standby). One fully-failed write tick maps to ~100ms of estimated outage regardless of writer count, so the cushion absorbs unrelated noise without tolerating a real outage.
- PrimaryHandoverPolicy (30s) for kill-primary-pod: an ungraceful failover interrupts writes for exactly one primary handover.
- UpgradeOutagePolicy (90s) for upgrade-documentdb: a cross-version rolling upgrade's graceful switchover coincides with the extension migration under live write load, so it gets its own, larger budget (calibrated from a real smoke run measuring ~33s, with headroom). The longer whole-topology restart is bounded by MustRecoverWithin, not the write-outage budget.

Operation runner modes + Long-Haul Smoke Gate

The operation runner now supports three modes (LONGHAUL_OPERATION_MODE): random (production long-haul — weighted, one op every 10s, global cooldown), sequence (an explicit ordered LONGHAUL_OPERATION_SEQUENCE, each op once, exits when the sequence completes/fails), and disabled. All modes keep the continuous writer/verifier workload active.

Sequence mode powers a new PR smoke gate — .github/workflows/longhaul-smoke.yml — that builds the operator/sidecar from the PR, stands up a real DocumentDB on kind, and runs the actual driver through scale-up → scale-down → upgrade-documentdb → kill-operator-pod → kill-primary-pod once each, asserting every op reaches PASSED, the run reaches COMPLETE, and the backup verifier scheduled+completed a snapshot. The upgrade-documentdb step is a real cross-version rolling upgrade between two published releases (0.110.0 → 0.113.0, both hardcoded and pulled from the public registry — the smoke builds no database image), exercising the operator's rolling-update path with clean semvers the image-rollback webhook accepts.

Design notes

Operator discovery. The operator Deployment name is a chart-pinned literal (documentdb-operator, singleton install), so it is a safe constant; the namespace is install-overridable, exposed as LONGHAUL_OPERATOR_NAMESPACE (default documentdb-operator). There is no back-reference from the DocumentDB CR to the operator, so cluster-derived discovery isn't viable. kill-operator-pod builds its pod selector from the Deployment's own spec.selector.matchLabels rather than a hardcoded label (the operator label is release-name dependent), and refuses to proceed on an empty selector so it can never list/delete every pod in the namespace.
Primary discovery. kill-primary-pod reads the CNPG Cluster.Status.CurrentPrimary (CNPG cluster name == DocumentDB CR name) to target the primary pod, then waits for the promoted primary to change and steady state to return.
HA guard. kill-primary-pod and upgrade-documentdb skip (do not fail) when instancesPerNode<2 — the disruption would be guaranteed downtime with no standby. Skips don't consume the scheduler cooldown and are re-evaluated on the next tick.

Changes

operations/kill_operator.go, operations/kill_primary.go (+ unit tests): the two chaos ops.
operations/{registry,runner,sequence}.go (+ tests) and operations/scheduler.go: the operation runner with random / sequence / disabled modes.
journal/policy.go, journal/journal.go: duration-based OutagePolicy, WritesPerSecond per window + EstimatedWriteOutage, SetWriteRate, and the shared NoOutagePolicy / PrimaryHandoverPolicy / UpgradeOutagePolicy helpers.
workload/writer.go: AggregateWriteRate(numWriters).
monitor/{health,k8sclient}.go: GetPrimaryInstance / DeletePod on ClusterClient; CNPG scheme registration.
operations/scale.go, operations/upgrade.go: adopt the shared policies (scale ops were previously over-lenient count budgets; upgrade gets its own 90s budget).
config/config.go: LONGHAUL_OPERATOR_NAMESPACE, LONGHAUL_OPERATION_MODE, LONGHAUL_OPERATION_SEQUENCE + validation.
report/{report,checkpoint}.go: disruption-window table gains an Est. Write Outage column; report exposes operation-status / operation-results / operation-aggregates.
cmd/longhaul/main.go: supply the write rate to the journal; wire up the operation mode; register both ops.
deploy/rbac.yaml: chaos-op RBAC (see below).
.github/workflows/longhaul-smoke.yml: the sequence-mode PR smoke gate.
docs/designs/long-haul-test-design.md / test/longhaul/README.md: README slimmed to a runner's guide; design doc gains the outage-budget, backup-verification, HA-precondition, and e2e-relationship rationale.
go.mod: promote cloudnative-pg to a direct dependency.

RBAC

Included in deploy/rbac.yaml: the driver ServiceAccount gets get/list on clusters.postgresql.cnpg.io + delete on pods (kill-primary), and, via a namespace-scoped Role/RoleBinding on LONGHAUL_OPERATOR_NAMESPACE, get on deployments + get/list/delete on pods in the operator namespace (kill-operator). Documented in test/longhaul/README.md.

Verification

cd test/longhaul
go build ./...   # clean
go vet ./...     # clean
go test ./...    # all packages pass

The Long-Haul Smoke Gate runs the full sequence (including the real 0.110.0 → 0.113.0 upgrade) green on kind.

documentdb-triage-tool · 2026-07-10T17:25:21Z

🤖 Auto-triaged by documentdb-triage-tool.

Applied: test, documentation, dependencies, enhancement
Project fields suggested: Component test · Priority P2 · Effort XL · Status In Progress
Confidence: 0.88 (mixed)

Reasoning

component from path globs (test, docs, dependencies); effort from diff stats (746+78 LOC, 23 files); LLM: Adds new chaos fault-injection operations (kill-operator-pod, kill-primary-pod) to the long-haul test suite, touching multiple files across the test infrastructure including health monitoring, k8s client, config, and operation implementations.

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Add the first two chaos fault-injection operations to the long-haul operation scheduler: - kill-operator-pod: deletes the operator pod via its Deployment selector and waits for the Deployment to become Available again. Asserts the CNPG data plane is unaffected by an operator restart (small write-failure budget). - kill-primary-pod: deletes the CNPG primary pod to exercise the automatic failover path, guarded by an HA precondition (instancesPerNode>=2). Both plug into the existing Operation interface and are selected by the weighted scheduler (one disruptive op at a time, cooldown + steady-state gate). Recovery is judged by the health monitor and workload verifier. Adds ClusterClient.GetPrimaryInstance (reads CNPG Cluster status.currentPrimary) and DeletePod, a LONGHAUL_OPERATOR_NAMESPACE config knob (default documentdb-operator), unit tests, and README operations + RBAC notes. Controlled failover was intentionally excluded: DocumentDB exposes no single-cluster manual switchover, so it would only test upstream CNPG. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

Introduce journal.NoOutagePolicy (and the NoOutageWriteFailureCushion constant) as the single budget for operations that keep the write path (client -> gateway -> primary) up throughout, so they must not cause write failures. Apply it to: - kill-operator-pod (control-plane fault; was an inline budget of 5), and - scale-up / scale-down, which only add/remove a standby replica — the primary is never touched, so their previous ad-hoc budgets (20 / 50) were too lenient and could mask a regression that disrupted writes. The cushion is a small non-zero value (5) that absorbs unrelated background noise without tolerating a real outage: at the default workload rate (~50 writes/s) it is well under a second of stray errors. Centralizing it lets the value be recalibrated against real long-haul runs in one place. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

Express OutagePolicy as a wall-clock write-outage duration (MaxWriteOutage) instead of a raw write-failure count. The journal converts the observed failure count into an estimated outage using the workload's aggregate write rate, so budgets no longer scale with LONGHAUL_NUM_WRITERS. - policy: OutagePolicy.AllowedWriteFailures -> MaxWriteOutage; DisruptionWindow gains WritesPerSecond + EstimatedWriteOutage(); NoOutageWriteFailureCushion -> NoOutageWriteOutageCushion (300ms). - journal: New() defaults to DefaultWritesPerSecond; SetWriteRate lets main.go supply the real rate; OpenDisruptionWindow stamps the rate. - workload: expose AggregateWriteRate(numWriters). - ops: kill-primary 30s, upgrade 45s (was 50/200 failures). - report: surface Est. Write Outage column. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

Document that MaxWriteOutage (data plane) and MustRecoverWithin (control plane) assert on orthogonal subsystems and fail independently, and that MustRecoverWithin is the only path that fails the run when the cluster never converges (op errors are logged, not scored). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

MustRecoverWithin bounds the managed database cluster's return to full topology (all pods Ready, CR Ready) — not the operator/control plane. Reframe as write-availability vs. full-topology recovery to avoid implying scale/kill-primary recovery involves the operator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

kill-primary-pod and upgrade-documentdb both interrupt writes for a single primary handover, so they now share journal.PrimaryHandoverPolicy (30s) instead of diverging (was 30s vs 45s). The 45s was inflated by conflating the rolling upgrade's whole-topology restart with its write-outage window; that longer restart is bounded by MustRecoverWithin instead. A graceful switchover (upgrade) is no worse than an ungraceful failover (kill-primary), which also pays a detection delay. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

deploy/rbac.yaml now exists on main (added by documentdb#413), so add the chaos operations' required verbs rather than leaving them as a follow-up: - kill-primary-pod: get/list on clusters.postgresql.cnpg.io and delete on pods, added to the longhaul-test Role in the target namespace. - kill-operator-pod: a separate longhaul-test-operator Role/RoleBinding in the operator namespace (documentdb-operator) granting get on deployments and get/list/delete on pods, since the operator runs outside the driver's own namespace. Update the README RBAC section to reflect that the verbs are now granted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 5 comments.

+	// Resolve the pod set from the Deployment's own selector so we don't
+	// depend on the release-name-derived "app" label value.
+	selector := labels.SelectorFromSet(dep.Spec.Selector.MatchLabels).String()
+	pods, err := k.clientset.CoreV1().Pods(k.namespace).List(ctx, metav1.ListOptions{LabelSelector: selector})


+	// Wait for the Deployment to reschedule and become Available again.
+	recoveryCtx, cancel := context.WithTimeout(ctx, k.recovery)
+	defer cancel()
+	return k.waitForDeploymentAvailable(recoveryCtx)


+func (k *KillPrimaryPod) Execute(ctx context.Context) error {
+	primary, err := k.client.GetPrimaryInstance(ctx)
+	if err != nil {
+		return fmt.Errorf("get primary instance: %w", err)
+	}
+	if err := k.client.DeletePod(ctx, primary); err != nil {
+		return fmt.Errorf("delete primary pod %s: %w", primary, err)
+	}
+
+	// Wait for CNPG to elect a new primary and the cluster to settle.
+	recoveryCtx, cancel := context.WithTimeout(ctx, k.recovery)
+	defer cancel()
+	return k.healthMon.WaitForSteadyState(recoveryCtx)
+}


+  name: longhaul-test-operator
+  namespace: documentdb-operator


+metadata:
+  name: longhaul-test-operator
+  namespace: documentdb-operator
+  labels:


Address PR documentdb#421 review: - kill-operator-pod: fail fast when the operator Deployment has no matchLabels selector, so SelectorFromSet can't produce an everything selector that lists/targets unrelated pods. - kill-operator-pod: after deleting the pod, wait until that specific pod (by UID) is actually gone before checking Deployment availability. Pod deletion doesn't bump ObservedGeneration, so status can stay Available and mask the restart otherwise. - kill-primary-pod: validate GetPrimaryInstance returned a non-empty pod name and guard the health monitor against nil before dereferencing. - rbac.yaml: note that the operator-namespace Role/RoleBinding must be kept in sync with LONGHAUL_OPERATOR_NAMESPACE. Add unit tests for the empty-selector and empty-primary guard paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

…/longhaul-chaos-ops Signed-off-by: Wenting Wu <wentingwu@microsoft.com> # Conflicts: # test/longhaul/config/config.go # test/longhaul/deploy/rbac.yaml

Introduce a without-replacement "coverage mode" so the CI smoke gate exercises the production random scheduler path while still guaranteeing every operation runs at least once. Adds LONGHAUL_OPERATION_COVERAGE and LONGHAUL_OPERATION_SEED config, seeded RNG + completion-driven Run loop in the scheduler, and switches longhaul-smoke.yml from sequence to random+coverage with a jq assertion on operation-aggregates. Includes the operations registry/sequence/runner refactor, journal and report updates, and accompanying unit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

…/longhaul-chaos-ops

…>0.113.0 upgrade Replace the random coverage/seed scheduler machinery with an explicit sequence mode for the long-haul smoke gate so the PR gate deterministically exercises each operation (scale-up, scale-down, upgrade-documentdb, kill-operator-pod, kill-primary-pod) exactly once, in order, and asserts every one reached PASSED with the run COMPLETE. Make upgrade-documentdb a real cross-version rolling upgrade: start the cluster on the most recent published release before the under-test version (0.110.0; 0.111/0.112 were never published) via setup-test-environment image overrides, convert the CR to version-based references, then move documentDBVersion forward to the chart's documentDbVersion. Both endpoints are clean semvers so the image-rollback admission webhook accepts the patch (the previous non-semver candidate tag was Forbidden by the webhook). Removes OperationCoverage/OperationSeed config, the scheduler's rng/coverage selection, and their tests; keeps plain random mode intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

A real cross-version rolling upgrade (0.110.0 -> 0.113.0) in the smoke gate measured a ~33s primary-switchover write outage, tripping the 30s PrimaryHandoverWriteOutage budget that upgrade-documentdb previously shared with kill-primary-pod. That shared budget assumed a graceful upgrade switchover is no worse than an ungraceful failover, but the upgrade switchover is heavier: it coincides with the extension version migration under live write load and the new primary must come up on the new image before accepting writes. Split the budgets: add UpgradeWriteOutage (90s) / UpgradeOutagePolicy and use it for upgrade-documentdb, leaving kill-primary-pod on the 30s failover budget. The value carries headroom over the observed ~33s (and CI variance) while still catching a gross multi-minute regression; the whole-topology restart stays bounded by MustRecoverWithin. Calibration backed by a real long-haul upgrade, as the prior heuristic comment invited. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

Sequence mode paces operations entirely via the steady-state and recovery gates (SequenceRunner never consults the cooldown); LONGHAUL_OP_COOLDOWN is a random-scheduler-only rate limiter. Setting it in the sequence-mode smoke ConfigMap had no effect, so remove it and note why to avoid implying it does something. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

- Trim 'RBAC for chaos operations' to the one actionable fact (kill-operator-pod needs cross-namespace access via LONGHAUL_OPERATOR_NAMESPACE); rbac.yaml grants the rest. Drop the verb-by-verb Role breakdown, which is design detail. - Simplify the outage-budget prose: remove internal type names (journal.*, OutagePolicy, AggregateWriteRate) and point to the design doc for exact policies, keeping the user-relevant concepts (~30s failover, ~90s upgrade). - Fix an outdated claim: the smoke upgrade is now a real cross-version upgrade (start on the last published release, upgrade forward to the under-test version), not a same-image second-tag retag. - Note LONGHAUL_OP_COOLDOWN is random-mode only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

The smoke's cross-version upgrade target was re-tagged from the build artifact's candidate documentdb/gateway images. Since the database images are never actually built (the under-test release is public), pull the released target images straight from the public registry instead, exactly like the 0.110.0 base override. This makes the smoke's intent explicit: it upgrades between two published database releases (0.110.0 -> under-test) and builds no database image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

Pin the upgrade target to DB_TARGET_VERSION (0.113.0) alongside the already hardcoded DB_BASE_VERSION, instead of deriving it from ext_image_tag. Both endpoints of the smoke's upgrade are now explicit, published releases pulled from the public registry, keeping the two versions co-located and easy to bump with the chart's documentDbVersion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

Both upgrade endpoints are now hardcoded (DB_BASE_VERSION=0.110.0, DB_TARGET_VERSION=0.113.0), so replace the misleading "under-test" phrasing that implied the target auto-follows the chart's documentDbVersion. Also fix comments that suggested the database images come from the build artifact: the base and target are pulled from the public registry, and documentdb-image-tag is only kept to satisfy the shared action's mandatory DB-image load. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

The README is the operator's guide for people running the long-haul harness; several blocks were design rationale, not how-to. Move them to the design doc and leave the README with what a runner needs: - Outage-budget specifics and the HA auto-skip rationale -> design (Operations). - Backup verifier internals (scheduling liveness, 3-schedule completion gap, retention-leak arithmetic, black-box oracle) -> design (Backup Verification). - The test/e2e comparison table, shared-code, and future opportunities -> design (Relationship to test/e2e/). The README now links into those design sections and keeps the concise, actionable summaries (prerequisites, RBAC, config table, operations catalog, CI-safety pointer). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

Copilot

🟡 Changes recommended

Runner termination, recovery gating, outage measurement, and smoke-test timing contain correctness issues.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 38/38 changed files
Comments generated: 7
Review effort level: Balanced

+	} else {
+		<-ctx.Done()
+		j.Info("main", fmt.Sprintf("test ending: %v", ctx.Err()))
+		<-opRunner.Done()


+func (w *DisruptionWindow) EstimatedWriteOutage() time.Duration {
+	if w.WritesPerSecond <= 0 {
+		return 0
+	}
+	return time.Duration(float64(w.WriteFailures) / w.WritesPerSecond * float64(time.Second))


+	// Wait for the Deployment to reschedule and become Available again.
+	return k.waitForDeploymentAvailable(recoveryCtx)


+	recoveryCtx, cancelRecovery := context.WithTimeout(ctx, r.recoveryTimeout)
+	err = r.steadyStateGate.WaitForSteadyState(recoveryCtx)
+	cancelRecovery()
+	if err != nil {
+		return fmt.Errorf("operation %s post-recovery steady-state gate failed: %w", op.Name(), err)


+              LONGHAUL_OPERATION_MODE: "sequence",
+              LONGHAUL_OPERATION_SEQUENCE: "scale-up,scale-down,upgrade-documentdb,kill-operator-pod,kill-primary-pod",


+> **HA topology required for upgrade / failover ops.** `upgrade-documentdb` and
+> `kill-primary-pod` auto-skip when `spec.instancesPerNode < 2` (no standby to
+> absorb writes). Run with `instancesPerNode: 2` (or `3`) to exercise them. The
+> skip is free — the next scheduler tick re-evaluates, so scaling up at any point


+// OutagePolicy bounds the write outage of an automatic failover. Killing the
+// primary interrupts writes until CNPG detects the loss and promotes a standby.
+// It shares the single-primary-handover budget with upgrade-documentdb (see
+// journal.PrimaryHandoverPolicy).


Address Copilot review feedback on the long-haul chaos harness: - kill_operator: after the targeted pod is gone, wait for a *replacement* pod (different UID) to reach Running+Ready before trusting the Deployment-level availability check. Deleting a pod does not bump the Deployment generation, so its status can still count the pre-deletion replica as Ready and report a false recovery. - health/sequence/scheduler: add HealthMonitor.InvalidateSteadyState and reset the steady-state epoch when opening a disruption window, so the post-operation recovery gate must observe a health sample taken after the disruption rather than being satisfied instantly by a stale steadySince. - docs: qualify HA auto-skip as random-mode behaviour (sequence mode waits then fails); fix outdated kill-primary OutagePolicy comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

Address two more Copilot review findings: - scheduler/main: a terminal operation failure in random mode now ends the run immediately (Scheduler.Run returns and closes Done; main reacts to it) instead of looping until MaxDuration, which is unbounded in production and meant a real failure never surfaced a verdict. - journal/policy/writer: replace the write-failure-count / write-rate outage estimate with a timestamp-based measurement. A synchronous InsertOne blocks for the full server-selection timeout and errors only once, so a 30s outage produced ~1 failure and an estimate of ~100ms — a false PASS on the core safety check. Writers now report every attempt outcome with its start time and the window measures the outage as first-failing-attempt -> first-subsequent-success (open outages measured to window end). Removes the now-unused write-rate plumbing (SetWriteRate/AggregateWriteRate). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

The retention pruner used a hardcoded 5m interval, so the smoke gate's "pruner pruned N docs" assertion depended on the sequence run happening to outlast the default first tick. Sequence mode exits on operation completion, which can be faster. Add LONGHAUL_PRUNE_INTERVAL (default 5m) plumbed through the config and the Pruner, and set it to 30s in longhaul-smoke.yml so a real prune reliably fires within the bounded window. Document the new knob in the runner README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dceb76cc-e7c1-40f2-92ad-3b6de007281c Signed-off-by: Wenting Wu <wentingwu@microsoft.com>

documentdb-triage-tool Bot added dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation enhancement New feature or request test labels Jul 10, 2026

WentingWu666666 changed the title ~~test: add kill-operator-pod and kill-primary-pod chaos operations to long-haul~~ test(longhaul): add kill-operator-pod and kill-primary-pod chaos operations Jul 10, 2026

WentingWu666666 marked this pull request as ready for review July 10, 2026 17:52

WentingWu666666 requested a review from xgerman as a code owner July 10, 2026 17:52

Copilot AI review requested due to automatic review settings July 10, 2026 17:52

WentingWu666666 requested review from alaye-ms and hossain-rayhan as code owners July 10, 2026 17:52

Copilot started reviewing on behalf of WentingWu666666 July 10, 2026 17:53 View session

Copilot AI reviewed Jul 10, 2026

wentingwu000 and others added 6 commits July 14, 2026 15:00

WentingWu666666 force-pushed the developer/wentingwu/longhaul-chaos-ops branch from ace474b to 40c62ce Compare July 14, 2026 19:00

WentingWu666666 requested a review from Copilot July 22, 2026 14:07

Copilot started reviewing on behalf of WentingWu666666 July 22, 2026 14:08 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

wentingwu000 and others added 2 commits July 23, 2026 13:13

Merge remote-tracking branch 'upstream/main' into developer/wentingwu…

e53b7f9

…/longhaul-chaos-ops Signed-off-by: Wenting Wu <wentingwu@microsoft.com> # Conflicts: # test/longhaul/config/config.go # test/longhaul/deploy/rbac.yaml

WentingWu666666 marked this pull request as draft August 21, 2026 19:27

wentingwu000 and others added 3 commits August 21, 2026 15:49

Merge remote-tracking branch 'upstream/main' into developer/wentingwu…

f28b4a8

…/longhaul-chaos-ops

wentingwu000 and others added 7 commits August 28, 2026 11:12

WentingWu666666 requested a balanced review from Copilot September 3, 2026 14:38

Copilot started reviewing on behalf of WentingWu666666 September 3, 2026 14:38 View session

Copilot AI reviewed Sep 3, 2026

View reviewed changes

wentingwu000 and others added 3 commits September 3, 2026 10:53

Navigation Menu

Uh oh!

test(longhaul): add kill-operator-pod and kill-primary-pod chaos operations - #421

test(longhaul): add kill-operator-pod and kill-primary-pod chaos operations#421
WentingWu666666 wants to merge 22 commits into
documentdb:mainfrom
WentingWu666666:developer/wentingwu/longhaul-chaos-ops

WentingWu666666 commented Jul 10, 2026 •

edited

Loading

Uh oh!

documentdb-triage-tool Bot commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// Wait for the Deployment to reschedule and become Available again.
		return k.waitForDeploymentAvailable(recoveryCtx)

		LONGHAUL_OPERATION_MODE: "sequence",
		LONGHAUL_OPERATION_SEQUENCE: "scale-up,scale-down,upgrade-documentdb,kill-operator-pod,kill-primary-pod",

Uh oh!

Conversation

WentingWu666666 commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Long-haul chaos operations + sequence-mode smoke gate

Chaos operations

Outage policy: duration-based, writer-count-independent

Operation runner modes + Long-Haul Smoke Gate

Design notes

Changes

RBAC

Verification

Uh oh!

documentdb-triage-tool Bot commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

WentingWu666666 commented Jul 10, 2026 •

edited

Loading