Operating an organisation

Check readiness, understand seat lifecycle, change configuration safely, retire seats and troubleshoot.

The deployment workflow

An infrastructure repository applies three stages in order. The example roots live in examples/:

Every saved plan, rendered plan, log and output goes to examples/out/ for review.

Checking readiness

bin/orgctl status --context kind-steadmesh --namespace steadmesh-example
bin/orgctl verify --context kind-steadmesh --namespace steadmesh-example --timeout 10m

status prints the current conditions, the seats and the representative endpoints. verify asks the controller for a completely fresh pass: it re-verifies connections and re-probes every seat. It exits non-zero unless the organisation is OperationalReady. A no-change terraform apply is not a health check, so run verify after every deploy.

Conditions

The conditions are evaluated in this order. The first one that is not True becomes the reason for OperationalReady.

A seat that is stopped but has passed its probe still counts as ready. The probe never creates business work and never sends messages to people.

IntegrationsDegraded is informational and is not part of OperationalReady. It is True while an optional connection, such as a work tracker, is failing. Agents keep working without it, and work publication catches up when it recovers. Which connections are optional is described under connections.

Seat lifecycle

Seats have technical execution states, separate from any project's business status:

The controller acts on these rules:

• Wake. A pending message for a stopped seat scales it to one replica. This usually takes a few seconds.
• Idle stop. With idle_policy = "warm_then_stop", a warm seat with no pending work is stopped once idle_timeout has passed since its last activity.
• Configuration change. When a seat's configuration revision changes, its Pod is replaced at a safe boundary: the runner finishes or interrupts the turn, checkpoints, then exits.
• Crash. The controller confirms the old Pod is gone, never force-deleting it, before the new runner's lease takes over. Stale writers are rejected by lease generation.

Changing an organisation

• Instructions or templates. The plan's resolved_seats shows which seats get a new configuration revision. Each execution records the revision it ran with.
• Grants. A revocation takes effect for the next tool call, before the apply reports ready. Information a model has already read cannot be un-read. Revocation stops future access only.
• Seats. Adding a seat creates a new identity. Removing one retires it gracefully (below). Renaming a seat key is a retirement plus a new seat. To carry data over, set adopt_from = "<old seat id>" on the new seat once the old one has retired.
• Harness. Changing harness_profile keeps identity, memory, workspace and messages. A native session that cannot be converted is replaced by a portable handoff, with a recovery note.

Retiring a seat

Removing a seat from the declaration never cuts it off mid-task. The apply returns at once, and the seat retires in three steps:

1. Retiring. The seat takes no new work. Sending it a message fails with unavailable and nothing is queued. Messages already waiting for it go back to their senders as not delivered, with the original quoted. Wake schedules and readiness probes stop. Its current turn, if any, carries on, and it keeps its tools, memory and sandbox access.
2. Wind-down. The seat gets one last message: a retirement notice. It finishes or pauses its work, saves a handoff with handoff.update, releases or hands over the work items it owns, and tells the seats it works with. This is bounded by the grace period, 10 minutes by default. When the period ends, the running turn is interrupted.
3. Retired. Once the notice turn is done, or the grace period ends, the platform retires the seat. Whatever it still holds is handed back:
   • Unfinished work items it owns are released to ready, with a log entry pointing at its handoff.
   • Messages it never handled are returned to their senders.
   • The organisation's representatives get a summary: whether it finished, its handoff, the released work items and the returned messages.
   Its lease is fenced, credentials delivered to its sandbox are revoked, and its personal memory store is retired. The controller then stops the Pod and removes the runtime, keeping the workspace volume.

Adding the seat back before it retires cancels the retirement. Messages already returned stay returned. The grace period is the platform's RETIREMENT_GRACE: the chart value retirement.grace, or seat_retirement_grace on the modules/platform Terraform module. 0s retires removed seats at once, and what they held is still handed back. Deleting the whole organisation retires every seat immediately.

# Watch a seat wind down
kubectl --context kind-steadmesh -n steadmesh-example get agentseats
kubectl --context kind-steadmesh -n steadmesh-example get agentseat <seat-name> -o jsonpath='{.status.retiringUntil}'

Agents can change the organisation only the way people do: through the infrastructure repository and its deployment workflow, and only if you grant them access to that repository. Seat credentials have no access to the Kubernetes API.

Administrative controls

These are infrastructure controls. Normal business steering goes through representatives.

# Stop a seat's execution without editing its prompt or workspace
kubectl --context kind-steadmesh -n steadmesh-example patch agentseat <seat-name> \
  --type merge -p '{"spec":{"adminSuspended":true}}'

# Resume it
kubectl --context kind-steadmesh -n steadmesh-example patch agentseat <seat-name> \
  --type merge -p '{"spec":{"adminSuspended":false}}'

# Inspect a seat
kubectl --context kind-steadmesh -n steadmesh-example get agentseat <seat-name> -o yaml
kubectl --context kind-steadmesh -n steadmesh-example logs <seat-name>-0

Seat object names look like seat-representative-sean-bd156bfb. Find them with kubectl get agentseats -l steadmesh.io/seat=<seat key>.

Retention and deletion

• Retiring a seat stops new deliveries, revokes its capabilities and removes its Pod. Its workspace volume and database records are kept, because the volume has no garbage-collecting owner.
• Destroying the organisation with data_retention = "retain" removes the runtime and revokes everything, but keeps volumes and records. With "delete", the volumes go too.
• External resources are never deleted. That covers Slack workspaces, Linear projects and accounts, because connections are ownership = "external".
• Deleting a namespace or database from outside the platform bypasses retention. Protect them in the infrastructure repository, and use a Retain reclaim policy for production storage classes.

Observability

Every component logs structured JSON. Each line carries organisation, seat, execution and message IDs, so one inbound message can be traced from acceptance, through wake and harness execution, to tool calls and the reply. The platform exports Prometheus metrics on :8080/metrics:

The console

The Steadmesh Console is an optional, read-only browser view of a running organisation. It is off by default. When it is off, nothing is deployed and the platform does not serve the console API. To turn it on, set enable_console in the platform stage and apply:

terraform -chdir=examples/platform apply -var enable_console=true
orgctl console --context kind-steadmesh     # forwards http://127.0.0.1:8090

Without orgctl, run kubectl --context kind-steadmesh -n steadmesh-system port-forward deploy/steadmesh-console 8090. The console listens only on its Pod's loopback interface and has no login of its own, so it can be reached only by someone allowed to port-forward in the control-plane namespace.

The console's ServiceAccount can only read AgentOrganizations, AgentSeats, Pods and PersistentVolumeClaims. The platform accepts that one identity on /console/v1, and every route there is a GET. Tool inputs and connector results shown in run details have credential-like fields (token, secret, password, API key, authorization) replaced with [redacted].

Rotating credentials

Connections refer to credentials by secret_ref: a Kubernetes Secret in the control-plane namespace (k8s:<name>) or a Vault KV v2 entry (vault:<path>). To rotate one, replace the value in place and keep the reference. You don't need to run Terraform or restart anything.

kubectl -n steadmesh-system patch secret slack-credentials \
  -p '{"stringData":{"bot_token":"xoxb-new","app_token":"xapp-new"}}'
vault kv put secret/steadmesh/linear api_key=lin_api_new

When a replacement is wrong. A new credential that the service rejects is never used. If the previous credential still works, it stays in use for credentials.grace (default 10 minutes). After that, or immediately if the previous credential is also rejected, the connection becomes unavailable and the reason is shown. Putting a valid value in the secret recovers it automatically.

When the secret is deleted, the previous credential stays in use for the grace period, and then the connection becomes unavailable. When the secret store can't be read, for example because Vault is sealed or unreachable, the current credential stays in use for credentials.maxStale (default 24 hours) while refreshes keep retrying.

Status. Each connection's credential state appears in the console's Readiness view and in the platform's verify response; orgctl status and orgctl verify show it in the ConnectionsAuthenticated condition message. The states are current, replacement_rejected, secret_missing, refresh_failing and unavailable, with the secret's version and the reason. While a required connection still works but is degraded, ConnectionsAuthenticated stays true with reason CredentialRefreshDegraded. No secret value is ever shown or logged.

Vault login. The platform's own Vault login is renewed before it expires, and it logs in again if renewal fails. That is separate from reloading the credentials stored in Vault. It is reported by the steadmesh_vault_login_ok and steadmesh_vault_login_expiry_seconds metrics.

Keeping values out of Terraform. Secrets that Terraform creates (the quickstart's slack_bot_token and similar) keep their values in that stage's state. To avoid that, create the Secrets or Vault entries yourself and pass only references in existing_secret_refs.

Troubleshooting

Steadmesh documentation. Source: docs/ in the repository.