Ops

Secrets

All secrets live in /etc/orchestra/secrets.env (mode 0600), loaded by every systemd unit via EnvironmentFile. A committed template is available at config/secrets.env.example — copy it and fill in real values.

Key	Used by
ANTHROPIC_API_KEY	ccr proxy (all claude invocations route through it)
ANTHROPIC_BASE_URL	set to http://127.0.0.1:3456 so the claude CLI uses the ccr proxy
ANTHROPIC_API_KEY_CCR	ccr proxy internal (set to the real API key; ccr strips the _CCR suffix)
GITHUB_APP_ID, GITHUB_APP_INSTALLATION_ID, GITHUB_APP_PRIVATE_KEY	GitHub MCP + webhook verify
GITHUB_WEBHOOK_SECRET	listener HMAC verify
GITHUB_BOT_USERNAME	listener filter (only dispatch review-requested-from-bot events); default orchestra
LINEAR_API_TOKEN	delegator/developer MCP + Linear status transitions
LINEAR_WEBHOOK_SECRET	listener HMAC verify
LINEAR_BOT_USER_ID	listener filter (only dispatch assign-to-bot events)
SLACK_BOT_TOKEN	dispatcher Slack notify
SLACK_SIGNING_SECRET	listener HMAC verify
SLACK_CHANNEL	dispatcher human_task notifications
SENTRY_WEBHOOK_SECRET	listener HMAC verify for Sentry webhooks
LANGFUSE_WEBHOOK_SECRET	listener HMAC verify for Langfuse webhooks

Paths (all env-overridable)

Env var	VM default
ORCHESTRA_REPO_DIR	/opt/orchestra
ORCHESTRA_DB	/var/lib/orchestra/state/orchestra.db
ORCHESTRA_WORKSPACES_DIR	/var/lib/orchestra/workspaces
ORCHESTRA_REPO_LOCK	/var/lock/orchestra-repo.lock

Provision (fresh VM)

# 1. Place repo at /opt/orchestra (rsync or git clone)
sudo ./scripts/provision.sh
# 2. Fill /etc/orchestra/secrets.env
sudo systemctl restart orchestra-listener orchestra-dispatcher

provision.sh installs Node 22, Caddy, the claude CLI + ccr, systemd units, the database schema, and a stub secrets file. It also enables orchestra-maintainer.timer for fresh VM provisions.

Deploy (existing VM)

There are three deploy paths depending on who authored the change. Agents may run only apply-local-by-agent.sh; apply-local.sh and deploy.sh are blocked for agents by pre_tool_gate.sh.

Laptop authoring (push → remote deploy)

git push origin main
ORCHESTRA_VM=user@host ./scripts/deploy.sh

deploy.sh requires ORCHESTRA_VM=user@host. It SSHes in, pulls origin/main, then hands off to scripts/apply-local.sh for the build/wire/restart/health steps. It exports CLAUDE_AGENT_NAME=deploy in the remote block so the auto-recover hard-reset still fires on divergence. deploy.sh is operator-only.

VM authoring (operator editing /opt/orchestra directly)

ssh user@host
orchestra shell --repo            # claude session, cwd = /opt/orchestra
# edit, commit, push from inside the session
sudo /opt/orchestra/scripts/apply-local.sh

apply-local.sh is idempotent — the same script deploy.sh calls. The pre-tool-gate hook recognises the invoking Unix user as an operator (listed in /etc/orchestra/operators) and allows main pushes, CLAUDE.md edits, and --force/--mirror pushes. Agents (anything with CLAUDE_AGENT_NAME set) remain restricted.

Agent deploy (builder)

The builder agent uses apply-local-by-agent.sh, the only deploy script the pre-tool-gate allows agents to run. It builds from the worktree, rsyncs artifacts to /opt/orchestra (using --delete to remove stale artifacts from the canonical checkout), and restarts orchestra-listener and orchestra-dispatcher only. It runs under a deploy lock.

Both deploy scripts enable all scheduled-agent timers on deploy — auditor, documenter, gc, maintainer, memoriser, and monitor — so timers are re-enabled on every redeploy even if one was disabled for any reason.

NODE_ENV handling: Both apply-local.sh and apply-local-by-agent.sh set NODE_ENV=development during npm ci and npm run build so devDependencies install despite the system-wide NODE_ENV=production. A build:ci package.json script is the standard build entry point for agents.

Post-deploy smoke tests: Both scripts run post-deploy smoke tests. The /health endpoint must return HTTP 200 (which includes a 4-table schema check), and the /metrics endpoint must return valid JSON — catching silent monitoring degradation before it goes undetected. apply-local.sh (operator path) also runs a broader suite: systemd unit syntax validation via systemd-analyze verify, service liveness checks (systemctl is-active for listener, dispatcher, and caddy), hook script syntax checks (bash -n), and a TUI bundle load check that verifies the compiled entry point imports without errors. Each test is non-fatal individually — warnings go to stderr and a summary count is printed.

Health-check verification: The builder's apply-local-by-agent.sh polls the listener health endpoint after restart. If 4xx/5xx responses persist or the service fails to start within 30s, the script automatically rolls back to the last known-good commit stored during the deploy sequence.

What gets restarted

Only orchestra-listener.service and orchestra-dispatcher.service restart during deploys. ccr.service is a routing proxy that doesn't need restarting on code changes, and restarting it kills in-flight LLM connections for any active agent session.

Operators

Operators are humans authorised to drive orchestra directly. Identity is the Unix user, registered in /etc/orchestra/operators (one username per line). bootstrap-vm.sh seeds the file with orchestra on a fresh provision; the orchestra Unix user is a system/service account. Humans SSH as their own account and are registered separately.

# 1. Create the Unix user (admin-side):
sudo useradd --create-home --shell /bin/bash <username>
sudo mkdir -p /home/<username>/.ssh && sudo chmod 700 /home/<username>/.ssh
# Drop the operator's pubkey into /home/<username>/.ssh/authorized_keys

# 2. Register them as an orchestra operator:
sudo /opt/orchestra/scripts/add-operator <username>

add-operator does four things:

• Adds the username to /etc/orchestra/operators (pre-tool-gate bypass).
• Adds them to the orchestra Unix group (workspace write access).
• Sets up .claude.json with ccr proxy configuration in the operator's home dir.
• Re-asserts ACLs on ${ORCHESTRA_WORKSPACES_DIR} (idempotent).

It prints SSH + VS Code onboarding hints for the operator's local machine.

Workspace permissions

${ORCHESTRA_WORKSPACES_DIR} (default /var/lib/orchestra/workspaces) is owned by orchestra:orchestra mode 2775 (setgid) with a default ACL of g:orchestra:rwX. Files created by the dispatcher (running as orchestra) inherit group orchestra with group-write, so operators in that group can edit them via VS Code Remote-SSH as their own Unix user without sudo or file-ownership gymnastics.

If setfacl ever drifts (e.g. someone runs chmod -R 755 by hand), re-run sudo bash /opt/orchestra/scripts/bootstrap-vm.sh — it reapplies the ACL idempotently.

Editing a workspace in VS Code (operator workflow)

Each operator needs an SSH alias on their local machine. Once add-operator has run, add this to ~/.ssh/config:

Host orchestra-vm
  HostName orchestra.example.com
  User <their-operator-login>

Then on the VM:

orchestra code                    # list available task workspaces
orchestra code STU-841            # auto-picks repo if there is only one
orchestra code STU-841 example-core
orchestra code --repo             # open /opt/orchestra itself

The script prints a clickable vscode://vscode-remote/ssh-remote+<host>/<path> URI. Most terminals turn URIs into Cmd/Ctrl-clickable links; clicking opens the operator's local VS Code with the Remote-SSH extension and attaches to the path on the VM. A CLI equivalent (code --remote ssh-remote+...) is printed alongside for non-clickable terminals.

SSH host detection priority: $ORCHESTRA_SSH_HOST → first site in /etc/caddy/Caddyfile → hostname -f. If the operator's SSH alias differs from the Caddy hostname (likely — Caddy serves the public HTTPS endpoint, SSH may use a different name), set ORCHESTRA_SSH_HOST=<alias> in the shell rc on the VM.

Bootstrap

VM bootstrap

bootstrap-vm.sh runs on every deploy (idempotent). It installs systemd units via glob-based install commands (orchestra-*.service, orchestra-*.timer), so new unit files in systemd/ are auto-discovered — no install-line changes are needed when adding a new agent. Beyond schema migrations and path creation, it runs git worktree prune to clear stale worktree metadata that accumulates when worktrees are removed by gc-workspaces.sh without the parent repo knowing. It enables orchestra-monitor.timer, orchestra-maintainer.timer, orchestra-auditor.timer, orchestra-memoriser.timer, orchestra-documenter.timer, and orchestra-gc.timer.

Local dev

./scripts/bootstrap-local.sh
# state under ./local/ — gitignored

Scheduled agents

Scheduled agents run through the dispatcher lifecycle. run-scheduled-agent.sh enqueues a pending task in mas_tasks (source=scheduled) instead of spawning claude directly. The dispatcher picks it up on the next poll tick, creates a git worktree under ${ORCHESTRA_WORKSPACES_DIR}/{plan_id}/, and spawns the agent. On completion the dispatcher handles status transitions and the GC timer reclaims the worktree.

Script signature: run-scheduled-agent.sh accepts <agent> <prompt> [repo] [--comment "text"]. The repo argument overrides the default orchestra for agents that operate on other repos. The optional --comment flag passes an operator-supplied comment through source_meta_json for TUI manual triggers, with proper JSON quoting to prevent injection.

Canonical config: The routing.yaml scheduled section is the canonical reference for what each agent runs — every entry carries prompt and repo fields. The TUI reads these for the manual trigger feature and run-scheduled-agent.sh passes them to the dispatcher.

Agent config validation: run-scheduled-agent.sh validates that the agent has a config file (.claude/agents/{agent}.md) before enqueuing a task. If the config is missing, the script exits with an error and does NOT write a mas_tasks row, preventing phantom task rows for misconfigured or missing agents.

Overlap policy: If a timer fires while a prior invocation of the same agent is still active, the script records a mas_tasks row with status=skipped and sets error to the overlap reason, making skipped runs visible in the DB and TUI. Workers in waiting_for_human status are excluded — workers paused on human questions do not block subsequent fires of the same agent; only actively running workers cause skips. The active-worker count also includes retry_requested workers (the dispatcher actively retrying a failed worker), which occupy the same agent slot, so a new invocation waits until the retry loop completes. The guard also checks mas_tasks for pending scheduled rows, closing a narrow race where a timer inserts a row before the dispatcher picks it up and a second timer firing in that window would otherwise create a duplicate.

Manual trigger: The TUI Scheduled panel includes a manual trigger button for each scheduled agent. When triggered, the TUI dispatches the agent with an optional inline operator comment via run-scheduled-agent.sh --comment. The gc timer's pipeline: [] is intentional — gc is infrastructure-only with no agent dispatched, so it is excluded from the manual trigger UI.

Timers:

• orchestra-auditor.timer — weekly, Fri 17:00 UTC (infrastructure security scan of the harness repo)
• orchestra-documenter.timer — daily, 21:00 UTC
• orchestra-gc.timer — daily, 03:00 UTC (no agent — infrastructure only)
• orchestra-maintainer.timer — twice daily, 06:00 and 22:00 UTC
• orchestra-memoriser.timer — daily, 20:00 UTC
• orchestra-monitor.timer — three times daily, 00:00, 08:00, 16:00 UTC

All agent timers include RandomizedDelaySec=60 to spread firings away from exact wall-clock alignment, reducing thundering-herd load on systemd at hour boundaries.

Monitoring

orchestra                            # TUI: sessions, human tasks, timers, logs (tabbed: journald, webhooks, sessions)
journalctl -u orchestra-listener -f
journalctl -u orchestra-dispatcher -f
journalctl -u ccr -f                 # ccr proxy logs (model routing decisions)
sqlite3 $ORCHESTRA_DB                 # direct inspection
curl http://127.0.0.1:3000/metrics   # operational metrics: latency percentiles, error rates per source

The dispatcher logs semaphore saturation changes at each tick — when the number of available permits changes, it prints [dispatcher] semaphore: N/M permits available. Use this to monitor worker concurrency pressure: journalctl -u orchestra-dispatcher -f | grep semaphore.

Listener operational metrics: The listener serves GET /metrics, returning JSON with total_requests (counters by source: github, linear, slack, sentry, langfuse, health, unknown), error_count (4xx/5xx by source), and latency (p50, p95, p99 over the last 100 requests). Metrics are collected in-memory via Fastify onRequest/onResponse hooks — no persistence in the listener itself. The /metrics endpoint is excluded from its own request counters so monitor scraping doesn't pollute the metrics. The monitor agent scrapes this endpoint three times daily and computes deltas against its previous snapshot stored in ${ORCHESTRA_STATE_DIR:-/var/lib/orchestra/state}/monitor-metrics-prev.json, alerting on elevated error rates (>5% per source with ≥10 requests) and sustained high latency (p95 > 2000ms).

GitHub setup requirements

• GitHub App installed on the your-org org (PR read/write, contents read, checks read)
• orchestra GitHub account — separate from the App identity; used as the review target to avoid a self-loop
• Webhook → https://orchestra.example.com/webhook/github

Ingestion

Linear

Field	Value
Endpoint	POST /webhook/linear
Signature	HMAC-SHA256 of raw body via linear-signature header w/ LINEAR_WEBHOOK_SECRET
Dedup key	${webhookId}-${webhookTimestamp} (Linear has no deliveryId field); seen_events table purged every 6h by listener + on dispatcher startup
Filter	type=Issue AND data.assigneeId == LINEAR_BOT_USER_ID AND (action=create OR action=update with assigneeId in updatedFrom)
source_meta_json	{ issue_id, issue_identifier, issue_url, team_key, team_id, creator_id, creator_name, creator_email }; creator fields resolved by the listener via the Linear API fetchIssueCreator() before delegator spawn
Event-loss prevention	The listener writes a placeholder mas_tasks row (status=pending, plan_json=null) before spawning the delegator, so every event has a visible task row even if the delegator crashes. The delegator uses UPDATE (not INSERT) — the row already exists. A listener-enforced timeout kills the delegator and marks the task failed if it doesn't complete.
Worker env	LINEAR_API_TOKEN, LINEAR_ISSUE_ID, LINEAR_ISSUE_IDENTIFIER, LINEAR_ISSUE_URL, LINEAR_TEAM_KEY, LINEAR_TEAM_ID
Reply: comments	Worker calls the GraphQL commentCreate mutation
Reply: state	Dispatcher transitions on lifecycle change (the worker doesn't touch state)
Status mapping	running → In Progress · waiting_for_human + failed → Blocked · done → Done; match by name first, fall back to Linear state type

GitHub

Field	Value
Endpoint	POST /webhook/github
Signature	HMAC-SHA256 via x-hub-signature-256 w/ GITHUB_WEBHOOK_SECRET
Dedup key	x-github-delivery header; seen_events table purged every 6h by listener + on dispatcher startup
Filter	event=pull_request AND action=review_requested AND requested_reviewer.login == GITHUB_BOT_USERNAME; additionally issue_comment.created with @orchestra in body, pull_request_review.submitted with @orchestra in the review body, and pull_request_review_comment.created with @orchestra in the inline comment body
Reply	Worker comments on the PR / opens reviews via the gh CLI; @orchestra mention events are acknowledged with a 👀 emoji reaction on the triggering comment

Slack

Field	Value
Endpoint	POST /webhook/slack
Signature	x-slack-signature + x-slack-request-timestamp w/ SLACK_SIGNING_SECRET
Dedup key	slack-${channel}-${event.ts}; seen_events table purged every 6h by listener + on dispatcher startup
Handles	url_verification handshake · block_actions (button → human_tasks resolution) · regular Events API · app_mention (routed to a direct answer or the developer agent for action requests) — non-app_mention events (message, reaction_added, etc.) are acked with 202 and logged without spawning a delegator
Reply	Bot posts via SLACK_BOT_TOKEN. Human_task questions render as Block Kit interactive blocks. Multi-question tasks use static_select menus + a Submit button for atomic answer collection; single-question tasks keep per-option buttons. Each select includes an "Other… (type below)" option with free-text input. Option descriptions render as markdown bullet points above the menus rather than in Slack's 75-char-limited description field. After answering, interactive blocks are removed and a Q&A summary with an "Answered!" marker is shown inline. The task creator is @-mentioned in a threaded reply for actual notification delivery — Block Kit context blocks don't trigger pings. The creator is resolved deterministically by the listener via the Linear API before forwarding to the delegator, using users.lookupByEmail exclusively. For Linear-origin tasks where the creator has a resolved Slack user ID, the notification is sent as a direct message instead of a channel post; DMs skip the threaded @-mention since the DM itself is the notification. Non-Linear tasks and unresolved lookups fall back to the team channel (SLACK_CHANNEL).

Sentry

Field	Value
Endpoint	POST /webhook/sentry
Signature	HMAC-SHA256 via sentry-hook-signature header w/ per-event secret; verified by verify/sentry.ts
Dedup key	sentry-${fingerprint}; seen_events table purged every 6h by listener + on dispatcher startup
Dispatches to	developer agent for error investigation
Filter	Errors grouped by fingerprint; deduplication prevents duplicate dispatches for recurring errors

Langfuse

Field	Value
Endpoint	POST /webhook/langfuse
Signature	HMAC-SHA256 via x-langfuse-signature header w/ LANGFUSE_WEBHOOK_SECRET; verified by verify/langfuse.ts
Dedup key	langfuse-${traceId}; seen_events table purged every 6h by listener + on dispatcher startup
Dispatches to	developer agent for observability event analysis

Scheduled

Field	Value
Source	systemd *.timer units (orchestra-auditor.timer weekly Fri 17:00, orchestra-documenter.timer at 21:00, orchestra-gc.timer at 03:00, orchestra-maintainer.timer at 06:00 and 22:00, orchestra-memoriser.timer at 20:00, orchestra-monitor.timer at 00:00, 08:00, 16:00 UTC — all invoke run-scheduled-agent.sh)
Bypass listener	Yes — the timer invokes run-scheduled-agent.sh, which enqueues a pending task in mas_tasks (source=scheduled). The dispatcher picks it up on the next poll tick.
Dedup	Not needed — timer firings are unique

Crash recovery

Dispatcher restart: reconciles running_workers against live PIDs. Dead PIDs → retry_requested (not failed), so dispatchRetries() auto-resumes them via --resume <session_id>. Pending mas_tasks are re-picked automatically, so there is no operator toil retrying orphaned workers after deploys.

Reserved-but-never-started workers — rows where the worker was reserved (running_workers inserted) but runWorker never reached attachRunningWorker (typically because the semaphore was full at dispatcher restart) — are detected by reconcileOnStartup. These rows have pid=null and workspace=null. They are re-queued as pending so the dispatcher spawns a fresh worker (rather than retry_requested, which dispatchRetries fails on because there is no session_id to resume).

Listener restart: reconciles orphaned placeholder mas_tasks rows — tasks with status=pending and a source_event_id older than 8 minutes (scaled alongside the delegator timeout). It also checks process aliveness — if a corresponding delegator process appears to still be running, the row is left untouched to avoid racing with an in-flight delegator. These orphaned tasks represent events whose delegator never completed (e.g. a VM crash during processing). The listener marks them failed with an error note so they are visible and retryable.

ccr proxy error recovery

Three layers of retry guard against ccr proxy synthetic model errors (model="<synthetic>", "Content block is not a text block"):

1. Listener-level: When the delegator session contains the error pattern, the listener clears the seen_events dedup entry and re-spawns the delegator up to 2 times. This prevents permanent event loss, since the dedup table would otherwise block Linear webhook retries.

2. In-process worker retry: When spawnWorker.ts detects the error pattern in worker stdout, it immediately re-spawns the worker with --resume — no need to wait for the hook+dispatcher poll cycle. A 500ms fast-poll sub-interval in the dispatcher provides a fallback if the in-process retry fails. The retry preserves the caller-supplied taskInput on the first attempt (containing the formatted human answer like [dispatcher] Human answered human_task ...), and only substitutes the generic auto-retry prompt on attempt > 1, so the human's answer is not lost.

3. Worker-level Stop hook: The on_stop_auto_retry.sh Stop hook detects the same error pattern in worker session output. Instead of marking the worker failed, it marks it for retry, and the dispatcher resumes the worker session. SQL sanitization prevents injection through error-message-derived values inserted into the database. It also detects synthetic errors in cumulative (resumed) transcripts via a parallel model='<synthetic>' + isApiErrorMessage=true path, since the terminal result event (is_error=true) exists only in per-attempt session files, not in cumulative --resume transcripts. The SQL status match spans IN ('failed', 'running') because the hook runs inside the claude process before the dispatcher marks the worker failed.

Workspace reclamation

gc-workspaces.sh runs daily via orchestra-gc.timer and after every apply-local.sh deploy. It removes finished-task workspaces older than a configurable threshold (default 3 days, env ORCHESTRA_WORKSPACE_RETENTION_DAYS). It skips workspaces for tasks still in running, waiting_for_human, or retry_requested status. It is dispatched by run-scheduled-agent.sh as a plain bash script (no agent).

The GC script uses a single loop that detects worktree vs. clone and uses the appropriate removal method:

• git worktrees (orchestra repo): git worktree remove
• product-repo clones: rm -rf

gc-workspaces.sh always runs git worktree prune unconditionally at the end of each run — regardless of reclaim count or dry-run mode — to ensure stale .git/worktrees metadata is cleaned up between deploys, since stale metadata can accumulate from sources other than the script's own removals (e.g. dispatcher teardown, deploy restarts).

Worker timeout

ORCHESTRA_WORKER_TIMEOUT_MS (default 600000 = 10 minutes) is set in the dispatcher systemd unit. The dispatcher enforces it in spawnWorker.ts — workers exceeding the timeout are killed and marked as failed.