Operations and rollback

Observe the difference between an approved decision, queued delivery, product acknowledgement, and effective access. The management API exposes scoped diagnostics so operators can identify lag without exposing private infrastructure in public documentation.

Check the system

GET /health reports a live management process. GET /ready checks expected schema tables and database access. Authorized operators can inspect GET /v1/operations/diagnostics and project-scoped diagnostics for failed or ambiguous work, worker observation, client drift, and lifecycle acknowledgement lag. GET /v1/jobs shows recent work status. Correlate a failed HTTP request with its X-Request-ID and redacted audit record.

Handle ambiguous work

Do not blindly repeat a provider write after a timeout or unknown response. Reconcile the saved provider reference and current target state, then use the reviewed retry or repair operation when its preconditions hold. For Mail, poll the stored message ID; a second POST can create a duplicate. For a product link, acknowledge only the exact installed version. For a lifecycle cursor, acknowledge only after the local transaction commits.

Rotate and restore

Rotate OIDC signing and lifecycle signing keys independently. Consumers need a coordinated key ID transition; an unexpected key must fail closed. Preserve encrypted credentials, old decryption keys for outstanding work, idempotency receipts, audit rows, and outbox entries through backup and restore. Prove that a restored product catches up from a signed snapshot before reopening access.

Contain a failed rollout

Disable the affected client or route, end local protected sessions, and hold new activation when its review or Mail path is incomplete. Keep the latest known revocations enforced. The old provider may remain available as a reviewed rollback artifact, but it must not silently restore access removed after cutover. Production rollback and emergency admission need private, independently reviewed runbooks and measured drills.