Data flow: Change apply

sequenceDiagram
  autonumber
  participant Admin as MSP admin
  participant Next as Next.js
  participant SbDB as Supabase Postgres
  participant ResyncEdge as graph-policies-read
  participant WriteEdge as graph-policies-write
  participant Graph as Microsoft Graph

  Admin->>Next: Apply now on approved change_request
  Next->>SbDB: verify status, preview TTL/result, approval identity
  Next->>SbDB: CAS dry_run_complete → applying + applying_started_at token
  Next->>ResyncEdge: invoke (source=pre_change)
  ResyncEdge->>Graph: GET all CA evidence (complete, beta policy shape)
  ResyncEdge->>SbDB: insert policy_snapshot source=pre_change
  Next->>SbDB: CAS-link pre_change_snapshot_id using apply token
  Next->>Next: re-evaluate break-glass + blast-radius safety
  Next->>WriteEdge: invoke { tenant_id, change_id, payload, claim token }
  WriteEdge->>WriteEdge: change-guard (claim, payload, snapshot, TTL, approval)
  WriteEdge->>Graph: GET live material state (+ ETag when exposed)
  WriteEdge->>Graph: PATCH/POST/DELETE (If-Match when available; no unsafe replay)
  Graph-->>WriteEdge: 204 No Content
  WriteEdge-->>Next: success, reconciled success, or outcome-unknown hold
  Next->>ResyncEdge: invoke (source=post_change)
  ResyncEdge->>SbDB: insert policy_snapshot source=post_change
  Next->>Next: verify approved effect in captured post snapshot
  Next->>SbDB: CAS applying → applied using apply token
  Next->>SbDB: insert audit_log
  Next-->>Admin: page revalidate, status=applied

State machine

Statuses are defined in lib/changes/status-guard.ts. Transitions below are the operator-visible paths; cancelled is available from several pre-apply states.

draft ── runDryRun ──┬──> dry_run_blocked ── re-run dry_run ──┐
                     │                                         │
                     ├──> awaiting_approval ── approveChange ──┤
                     │         (second admin; creator cannot   │
                     │          self-approve when required)     │
                     │                                         v
                     └──> dry_run_complete ─────────────────────┘
                                    │
                    ┌───────────────┼───────────────┐
                    │               │               │
            (future scheduled_for)    │         approveAndApply
                    │               │         (or cron when due)
                    v               v               v
              stays complete    cancelled      applying
              until due                          │
                    │                            ├── pre_snapshot_failed → failed
                    │                            ├── graph write error → failed
                    │                            └── success → applied
                    │                                      │
                    └──────── cron picks up ───────────────┘
                                                         │
                                              rollback (applied only)
                                                         v
                                                   rolled_back

Status reference

Dry-run can be re-run from: draft, dry_run_complete, awaiting_approval, dry_run_blocked, and failed. Re-running clears approval stamps and scheduled_for, and recomputes the next status from the fresh result.

Approval gate

When msp.require_approval is true or the dry-run marks a critical/destructive change, runDryRun lands in awaiting_approval instead of dry_run_complete. approveChange (second admin, not the creator) moves the row to dry_run_complete. approveAndApply rejects awaiting_approval directly so approval cannot be skipped.

Dry-run TTL

Immediate dry-run results expire after 30 minutes. A row with an explicit scheduled_for uses the separate seven-day scheduled preview TTL. Both values (DRY_RUN_TTL_MINUTES and SCHEDULED_DRY_RUN_TTL_MINUTES) are shared by the app and Edge guard; stale previews cannot apply until re-run.

applying and compare-and-swap

Before any Graph write, the app (or scheduler) atomically updates dry_run_complete → applying and stores applying_started_at. That exact value is carried through every later database CAS and into the Edge change guard, so a worker that resumes after a timeout cannot act under a newer claim. Only one caller wins; others get change_apply_conflict. A complete pre-change snapshot must then be linked while the caller still owns the token. If capture/link or the fresh safety recheck fails, Graph is not called.

The orphan cleanup uses applying_started_at, not dry_run_at; this prevents a newly claimed seven-day scheduled preview from looking like a stale worker. An apply held past the recovery lease is moved to failed with an operator-review marker, never silently returned to the apply queue.

failed recovery

A row in failed retains the last dry-run payload and error. Recovery:

1. For a definitive pre-write or Graph rejection, re-run dry-run (transitions back through blocked/approval/complete as appropriate).
2. Apply again once status is dry_run_complete and TTL is fresh.

Do not re-dry-run a row marked write_outcome_unknown_needs_review, orphaned_applying_needs_review, or operator_marked_failed. Graph may already contain the mutation. Create attempts retain server-owned reconciliation metadata and may never issue a second POST.

If Graph succeeded but post-change capture or verification failed, status stays applied with a verification marker and a critical operational alert. The change is live; rollback remains fail-closed until the required evidence is available.

Change-guard on graph-policies-write

Every CA write through graph-policies-write calls validateChangeForWrite (supabase/functions/_shared/change-guard.ts) before touching Graph. This is the authoritative server-side gate; the edge function does not trust the caller payload alone.

Apply mode requires:

• Row exists and caller payload matches stored change_request.payload (anti-tamper).
• status === 'applying' and the caller owns the exact applying_started_at claim.
• pre_change_snapshot_id is present and linked before Graph.
• dry_run_result.ok === true and dry_run_at within TTL.
• Required approval is fresh and comes from a different persisted creator.

Rollback mode requires:

• status === 'applied'
• Required snapshot ids match the caller
• The caller owns the exact rollback_in_progress:<timestamp> marker

Mismatch or wrong status returns 403 / 409 with generic codes such as change_not_applicable, payload_mismatch, or dry_run_stale.

Scheduled apply (cron)

Changes with a future scheduled_for remain at dry_run_complete until due. The changes-scheduled-apply edge function (pg_cron every 5 minutes) finds rows where status = 'dry_run_complete', scheduled_for is set, and scheduled_for <= now(), then mirrors approveAndApply:

• Re-checks preview TTL/result and distinct-admin approval when required
• Compares live material fingerprints to the approved baseline (portal drift)
• Re-runs break-glass and blast-radius inputs and fails closed when required evidence is unavailable
• CAS-claims applying, links a complete pre-snapshot, writes through the guarded Edge path, and verifies a complete post-snapshot
• Reconciles lost responses; ambiguous outcomes are review-held and non-idempotent creates are never blindly replayed
• Per-row error isolation - one failure does not block others

See docs/admin/scheduled-changes.md for operator-facing detail.

Why pre + post snapshots

• Pre-change snapshot is the rollback source. If the PATCH succeeds but produces unexpected behaviour, rollbackChange reads this snapshot's policy doc and PATCHes back to that shape.
• Post-change snapshot is the verification source. It confirms what Microsoft actually accepted (sometimes Graph's PATCH does normalization or drops fields silently - we want to record what landed, not what we sent).

Snapshots are full per-tenant captures, not policy-level. That's deliberate - it means a change can be rolled back even if Microsoft normalized fields we didn't touch.

Rollback

Interactive rollback (rollbackChange in lib/changes/service.ts):

1. Requires status === 'applied' and the snapshot evidence required by that kind.
2. Loads the linked post snapshot, linked pre snapshot (when applicable), and live object. Missing/unreadable evidence fails closed.
3. Proceeds only when live material state exactly matches the captured post-apply state. If live already matches the pre-change state (or an applied create is already absent), it reconciles without another write; any other state is divergence and blocks rollback.
4. Claims the row with an owned rollback_in_progress:<timestamp> marker.
5. Invokes graph-policies-write with mode: 'rollback'; the Edge guard validates the exact marker, compares the live material fingerprint, and adds If-Match only when Graph exposes an ETag.
6. On Graph success, CASes its own marker to rolled_back and audits.
7. Post-rollback resync: refreshes tenant evidence. Failure or truncation leaves the true status rolled_back but records a verification marker and raises an operational alert.

Non-reversible kinds (policy.delete, location.create, location.delete) and missing create IDs are rejected before Graph. policy.create rollback deletes the created policy via a synthesized delete payload.

Rollback constraints

• Cannot rollback if status ≠ applied
• Cannot rollback an existing-object edit without readable linked pre and post evidence
• Cannot rollback policy.create without its persisted created id and readable post evidence
• Cannot rollback when live state is unreadable or differs from both the captured post state and the already-restored pre state
• A rollback Graph failure leaves status at applied with error_message populated; admin can retry

Why immediate apply does not always require a second admin

Product decision for the default path. The dry-run requirement + the 30-minute TTL + the pre/post snapshots + the audit log are the safety net. Workspaces with require_approval or critical changes use the awaiting_approval second-admin step instead. A customer's compliance regime demanding standing four-eyes review on every change is covered by that flag plus critical-change promotion.