Architecture

Topology

Internet → Caddy (TLS) → listener (Fastify :3000)
                              │ write placeholder mas_tasks row
                              │ spawn delegator agent
                              │ onRequest/onResponse hooks for /metrics collection
                              ▼
                         mas_tasks (SQLite)
                              │ poll
                              ▼
                        dispatcher (daemon)
                              │ spawn workers via ccr proxy
                              ▼
              /var/lib/orchestra/workspaces/{task}/{repo}/
                    worker (claude -p --agent X)
                              │ PR → GitHub
                              ▼
                    reviewer (via review_requested webhook)

GitHub @orchestra mention ──► listener (via issue_comment / review webhook)
Slack app_mention ──► listener (via POST /webhook/slack) → delegator → developer
Sentry / Langfuse ──► listener (via POST /webhook/sentry|langfuse)
Monitor ──► GET /metrics (latency + error rates) → monitor-metrics-prev.json delta analysis

human_tasks table ──► dispatcher ──► Slack bot (notify)
Slack button click ──► listener ──► human_tasks (resolve)
                              │
                        dispatcher resumes worker

All claude CLI invocations route through ccr (Claude Code Relay) at http://127.0.0.1:3456. ccr sits between the claude CLI and the Anthropic API, enabling model swapping without changing agent definitions.

Operator interactive sessions use slash commands defined in .claude/commands/. These invoke specific agents in the operator's own terminal session — they bypass the listener/dispatcher pipeline entirely.

GitHub @orchestra mentions in issue comments, PR reviews, and inline review comments are handled by the listener, which spawns delegators that dynamically route to the developer or reviewer agent based on comment content. An 👀 emoji reaction is added to the triggering comment as an acknowledgement.

Sentry and Langfuse webhook listeners extend the inbound pipeline to observability tools — Sentry alerts are deduplicated by error fingerprint and dispatched to the developer agent for investigation; Langfuse trace webhooks feed observability events into the system for analysis.

Agents

Nine agent roles run on the system. Most are single-repo agents locked to the orchestra harness repo; the developer is a repo-agnostic product agent.

Agent	Trigger	Role
`delegator`	every inbound event	routes the event to the target agent(s), acknowledges on Slack/GitHub, and writes a dispatch plan into `mas_tasks`
`maintainer`	twice daily 06:00, 22:00 UTC	reviews the roadmap and generates evolutionary proposals; sole proposer of architecture-level changes
`builder`	event-driven	applies approved maintainer proposals and self-deploys
`documenter`	daily 21:00 UTC + maintainer proposals	keeps `docs/` in the orchestra repo current
`memoriser`	daily 20:00 UTC + maintainer proposals	promotes learnings into the memory layout (`memory/`, `CLAUDE.md`)
`monitor`	00:00, 08:00, 16:00 UTC	proactive health checks; silent when healthy, opens human_tasks on issues
`reviewer`	`pull_request.review_requested` + @-mentions	validates PRs against team and repo conventions
`auditor`	weekly Fri 17:00 UTC	infrastructure security scan of the orchestra harness repo only
`developer`	event-driven (optional)	example product-repo feature/bug agent, operating on `example-*` repos

Code organization

A shared/ directory contains TypeScript code shared between services (dispatcher and TUI) via TypeScript project references and Node.js subpath imports (#shared/*). The first shared module is agent_routing.ts — a single source of truth for routing maintainer proposals to agents by affected_paths. Both the dispatcher and TUI consume it. scripts/apply-local-by-agent.sh syncs shared/dist/ to the canonical checkout on deploy. A second shared module, eventSummary.ts, holds summarization logic (JSONL parsing into display-friendly structs) for the Sessions and Logs tabs — kind classification, AI message detection, and tool-call summarization are centralized there.

Test infrastructure: A vitest-based test suite provides smoke tests covering the three core services:

services/dispatcher/src/__tests__/planSchema.test.ts — validates the workerPlanSchema and planSchema Zod schemas (valid plans, validation rejections, default task_input, repo format, branch naming convention, and multi-worker plans)
services/listener/src/__tests__/health.test.ts — smoke test for the listener health endpoint
shared/__tests__/agent_routing.test.ts — smoke tests for the routing logic shared between dispatcher and TUI

Vitest is configured via vitest.config.ts at the repository root.

Processes

Unit	Type	Role
`orchestra-bootstrap`	oneshot	first-boot provisioning of the VM
`orchestra-listener`	long-running	receive + verify webhooks, spawn delegator; purges `seen_events` every 6h; serves `/health` (with 4-table schema verification) and `/metrics` (latency percentiles + error rates per source — the `/metrics` endpoint is excluded from its own counters; scraped by the monitor agent)
`orchestra-dispatcher`	long-running	poll queue, spawn/supervise workers, run scheduled agent ticks, route maintainer proposals; logs semaphore saturation changes (permits available / max) each tick
`ccr`	long-running	Claude Code Relay — proxy between the claude CLI and the Anthropic API for model swapping; NOT restarted during deploys, to avoid killing in-flight LLM connections
`caddy`	long-running	TLS termination
`orchestra-maintainer.timer`	twice daily 06:00, 22:00 UTC	roadmap review and proposal generation; dispatched via `run-scheduled-agent.sh`; includes a 60s randomized delay
`orchestra-memoriser.timer`	daily 20:00 UTC	memory promotion; dispatched via `run-scheduled-agent.sh`; includes a 60s randomized delay
`orchestra-documenter.timer`	daily 21:00 UTC	self-documentation — updates `docs/` in the orchestra repo; dispatched via `run-scheduled-agent.sh`; includes a 60s randomized delay
`orchestra-monitor.timer`	00:00, 08:00, 16:00 UTC	health checks: queue depth, worker liveness, systemd health, disk space, DB size; silent when healthy, opens human_tasks for issues; for automatable findings, offers handoff options to delegate cleanup to the builder agent; includes a 60s randomized delay
`orchestra-auditor.timer`	weekly Fri 17:00 UTC	infrastructure security scan of the orchestra harness repo: command injection, credentials, unsafe file ops, path traversal, sudo, network hardening; includes a 60s randomized delay
`orchestra-gc.timer`	daily 03:00 UTC	reclaim finished worker workspaces; dispatches no agent (infra-only); handles both worktrees and legacy clones

Workers are short-lived child processes of the dispatcher. They are not systemd-managed.

Data flow

Webhook arrives → listener verifies HMAC → dedup via seen_events
Linear assign-to-bot events → listener pre-resolves the issue creator via the Linear API → writes a placeholder mas_tasks row (status=pending) → spawns the delegator via dispatchToDelegator
GitHub @orchestra mention events (issue_comment.created, pull_request_review.submitted, pull_request_review_comment.created) → listener spawns a delegator that dynamically routes to the developer or reviewer based on comment content; acknowledges with an 👀 emoji reaction
Sentry alert / Langfuse trace webhooks → listener verifies webhook signatures, deduplicates, and dispatches to the developer agent
Slack app_mention events → listener filters via shouldDispatchSlack() — only app_mention events are dispatched; all other Slack event types (message, reaction_added, etc.) are acked with 202 without spawning a delegator — then routes to the delegator, which dispatches to the developer agent
The delegator determines the target repo using agent scope first (single-repo agents like builder, memoriser, documenter, maintainer, and auditor are locked to orchestra; repo-agnostic agents like developer use the wildcard example-*), then falls back to explicit directives, GitHub-sync links, or ticket content. It writes plan_json into the existing mas_tasks row via UPDATE, also resetting status='pending' and error=NULL to close the race window between listener placeholder creation and delegator plan writing. It uses envelope.task_id exactly when present — never inventing a different id — and verifies the UPDATE with a SELECT after write to catch silent no-ops, with a status-repair step that resets failed back to pending if the dispatcher raced ahead. A 300s timeout is enforced by the listener; on timeout the delegator session is preserved for resume and the timeout is raised to 30m. Failed/done tasks are revived to pending on re-dispatch so retries don't collide with the dedup table.
Dispatcher polls → validates the plan → spawns workers → updates status. Workers get in-process retry on ccr proxy synthetic errors via --resume, without waiting for the hook + dispatcher poll cycle.
Worker opens a PR and requests review from orchestra
pull_request.review_requested re-enters at step 1 → reviewer agent
Human tasks: worker writes a row → dispatcher notifies Slack (DMs for Linear tasks with a resolved Slack user, channel posts for all others) → human answers via the TUI → dispatcher either resumes the worker or creates a handoff task for a target agent
Maintainer proposals: maintainer writes a human_task → operator approves (or the dispatcher auto-approves low-risk proposals) → dispatcher inspects affected_paths and routes to the documenter (docs/ only), the memoriser (memory/ / CLAUDE.md only), or the builder (everything else) using the single source of truth in shared/agent_routing.ts

The listener runs a seen_events purge every 6h, deleting rows older than 1 day. This keeps the dedup table from growing unboundedly between dispatcher restarts.

Listener startup reconciliation: On startup, the listener scans for orphaned placeholder mas_tasks rows — tasks in status=pending with a source_event_id older than 5 minutes. These represent events whose delegator never completed (e.g., the VM crashed mid-processing). The listener marks them failed so they are visible in the TUI and retryable, rather than sitting silently in pending.

Task revival on re-dispatch: When the listener receives a webhook event that maps to an existing mas_tasks row (same source_event_id), it revives rows in failed or done status back to pending before re-spawning the delegator, so re-dispatches of previously completed events don't collide with the seen_events dedup table and get silently dropped.

Delegator session preservation: When the delegator times out, the listener preserves the session ID rather than treating it as a hard failure. This lets the dispatcher resume the delegator session later. The delegator timeout is 300s, and 30m on resume, to accommodate complex events that require multiple MCP tool calls.

TUI

The TUI (orchestra tui command) is a tabbed terminal interface built with Ink (React for terminals). Tabs: Sessions, Human Tasks, Timers (Scheduled + Maintenance), Artifacts, Logs (Journald / Webhook Events / Delegator Sessions), Stats, Team Memory, Skills.

Session rendering: AI assistant messages in the Sessions and Logs tabs render in full without truncation — operators can read through a worker session's AI reasoning directly in the TUI. All other event types (tool calls, user messages, tool results) remain truncated for scanability. The Logs tab's Delegator Sessions sub-panel is selectable with Enter (↵); pressing Enter opens a full transcript view of the selected session. The summarization logic (summarizeJsonLine, kind classification) lives in the shared services/tui/src/eventSummary.ts module consumed by both the Sessions and Logs tabs.

Maintenance panel

The Maintenance panel (under the Timers tab) shows maintainer proposals with an 80/20 layout: a list of proposals on the left (with scroll navigation) and a detail view on the right. Proposals display an age column so the operator can identify stale proposals at a glance. Custom text input supports proposal filtering and operator notes directly in the terminal. A status filter toggle is available — press f to cycle through filter modes: a (all), o (open), p (pending), d (done). The filter row shows the current selection highlighted, with a filtered/total count displayed in yellow when a filter is active. Auto-approved proposals display an [auto] tag: cyan for open proposals, a yellow ⚡ for pending, and a dimmed [auto] for terminal proposals (done or failed) so their auto-approve history is visible even after resolution.

Scheduled panel

The Scheduled panel (under the Timers tab) shows systemd timer status for scheduled agents. A monitor health badge at the top shows "Healthy" (green) when there are zero open monitor human_tasks, or a red alert count when issues need attention. Each timer entry includes the last-run status from the database — done (green), skipped (yellow), failed (red), or running (blue) — so the operator can see at a glance whether each scheduled agent is completing normally. The panel includes the weekly auditor (Fri 17:00).

Manual trigger: The Scheduled panel supports manually triggering any scheduled agent with an optional inline operator comment. The TUI reads agent prompt and repo configuration from routing.yaml's scheduled section (the canonical reference for what each scheduled agent runs). When the operator triggers a run, the TUI dispatches it through the same run-scheduled-agent.sh script (via the --comment flag), passing the operator's comment as source_meta_json metadata. The services/tui/src/routing.ts module parses routing.yaml line-by-line to extract scheduled configs — entries without a prompt (e.g., the gc timer, which dispatches no agent) are silently skipped from the trigger UI. Scheduled entries carry their prompt and repo fields in routing.yaml.

Artifacts panel

The Artifacts tab surfaces per-session markdown summaries written by scheduled agents into agents/<agent>/artifacts/<plan_id>.md. It lets the operator see what happened in each scheduled run (documenter, memoriser) without inspecting raw session transcripts.

Team page

The Team Memory page has scroll support for the file viewer, enabling navigation through longer memory files that exceed the terminal height.

VM

A small single VM: ~2 vCPU / 4–8 GB RAM / ~40 GB disk
Ubuntu 24.04 LTS
State survives reboots via SQLite on a persistent volume
Concurrency cap: 5 workers (env ORCHESTRA_MAX_WORKERS)
Workspace reclamation: orchestra-gc.timer runs gc-workspaces.sh daily, removing workspaces for completed tasks older than a configurable threshold (default 3d, env ORCHESTRA_WORKSPACE_RETENTION_DAYS). It uses git worktree remove for orchestra worktrees and rm -rf for product-repo clones, also runs after every apply-local.sh deploy, and unconditionally runs git worktree prune at the end of each run to clean up stale worktree metadata from any source.
Worker timeout: ORCHESTRA_WORKER_TIMEOUT_MS (default 600000 = 10 min), enforced by dispatcher spawn logic.
Deploy: only orchestra-listener and orchestra-dispatcher are restarted — ccr.service is NOT restarted, as that kills in-flight LLM connections. Agents use apply-local-by-agent.sh under the deploy lock; apply-local.sh and deploy.sh remain for operator use only. The agent deploy script uses rsync --delete to remove stale artifacts from the canonical checkout. Post-deploy smoke tests verify the listener health endpoint returns HTTP 200; the health endpoint verifies the presence of all four core schema tables (mas_tasks, running_workers, human_tasks, seen_events) before reporting healthy, returning 503 if any table is missing. If the health check fails (4xx/5xx or a failure to start within 30s), the builder auto-rolls back to the last known-good commit.
Systemd unit discovery: new .service and .timer files in systemd/ are auto-discovered by glob-based install -m 0644 commands in both apply-local.sh and bootstrap-vm.sh (systemd/orchestra-*.service and systemd/orchestra-*.timer). No install-line changes are needed when adding a new agent — only the systemctl enable --now list needs a new entry for the timer.

Paths

Repo: /opt/orchestra
State and workspaces: /var/lib/orchestra (DB at /var/lib/orchestra/state/orchestra.db)
Secrets: /etc/orchestra
Environment variables are prefixed ORCHESTRA_*.
GitHub org: your-org; product repos glob example-*; the self/harness repo is named orchestra.
Public hostname: orchestra.example.com (TLS terminated by Caddy).

Architecture ​

Topology ​

Agents ​

Code organization ​

Processes ​

Data flow ​

TUI ​

Maintenance panel ​

Scheduled panel ​

Artifacts panel ​

Team page ​

VM ​

Paths ​