Decisions
Architectural rationale for orchestra — why the system is shaped the way it is, grouped by concern. This is reference for anyone adopting or extending the system, not a changelog.
Architecture
| Decision | Rationale |
|---|---|
| Route all claude invocations through the ccr (Claude Code Relay) proxy | Enables model swapping per agent without changing agent definitions. ccr runs as a systemd service; the claude CLI points at it via ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY_CCR. Operators and agents transparently get model routing. |
| Single VM, no orchestration layer | Complexity budget. One machine is sufficient; shard later if needed. |
| SQLite as the unified queue | Atomic writes, dedup via a UNIQUE constraint, no extra process. WAL mode handles concurrent reader/writer safely. |
| Dispatcher polls SQLite (not event-driven) | Simplicity. A ~2s poll latency is acceptable and avoids SQLite trigger/notify complexity. |
| Caddy for TLS termination | Zero-config Let's Encrypt. No nginx config to maintain. |
| Worker concurrency cap = 5 | API rate-limit headroom plus VM RAM budget. Configurable via ORCHESTRA_MAX_WORKERS. |
| Scheduled agents run through the dispatcher lifecycle | run-scheduled-agent.sh enqueues a pending task rather than spawning claude directly; the dispatcher picks it up on the next tick. One lifecycle (workspace creation, timeout, crash recovery) for scheduled and event-driven agents alike. |
Listener pre-creates a placeholder mas_tasks row before spawning the delegator | Prevents silent event loss. Every seen event has a visible task row even if the delegator crashes before writing the plan; the delegator fills it in via UPDATE. A listener-enforced timeout marks the task failed if the delegator never completes. |
| All agent writes to the harness land via PR + squash-merge | Memoriser, documenter, and builder use the same gh pr create + gh pr merge --squash flow instead of direct push. GitHub's merge serialization replaces a push lock. Only operators push directly to main. |
Agents
| Decision | Rationale |
|---|---|
The delegator is the authoritative router; routing.yaml agent selection is a hint | Static routing tables break on edge cases. The delegator has full event + MCP context to route better. |
| Controlled autonomy — humans decide, agents execute | Trade-offs, preferences, and risk are human judgment calls. Agents escalate structured questions; once answered they execute precisely without further confirmation. |
| Single-writer ownership of auto-maintained paths | The documenter is the sole writer to docs/; the memoriser is the sole writer to CLAUDE.md and memory/. pre_tool_gate.sh enforces this at the tool-call level, preventing conflicts and guaranteeing a single source of truth. |
| The documenter watches the harness repo itself, not an external docs repo | The system's documentation lives alongside its code; no separate repo sync. A daily timer keeps docs current. |
| Maintainer + builder pair for roadmap-driven change | The maintainer reviews codebase and roadmap and proposes changes via human_task; on approval the dispatcher routes to documenter, memoriser, or builder by affected_paths. Separates "what to change" from "how to execute". |
Maintainer proposals routed by affected_paths | docs/ only → documenter; memory/ or CLAUDE.md only → memoriser; everything else → builder. The maintainer does not hardcode a target agent. |
Worker resumes via claude --resume <session_id> after a human_task | Preserves conversation context; avoids re-reading files from scratch. |
| Review is requested from a separate bot account, not the GitHub App identity | The App reviewing its own PRs is a self-approval loop; a separate account avoids it. |
| Human handoff uses a structured picker payload, not a free-text question | The TUI renders it as a native picker, and the structured answer is easy for the agent to parse on resume. |
| The listener resolves the Linear issue creator deterministically via API | LLM-based extraction missed fields. The listener fetches the creator via the Linear GraphQL API before forwarding; the value is stored in source_meta_json for Slack @-mention resolution. |
| The delegator is banned from opening human_tasks | It is a one-shot agent with no persistent session, so any human_task it opened would be orphaned. Ambiguous events write a failed mas_task instead. |
Agent-to-agent handoff via a handoff field on human_task options | When an option carries a handoff definition, the dispatcher creates a new mas_task for the target agent instead of resuming the delegating agent — e.g. the monitor delegating an automatable fix to the builder. |
| A dedicated monitor agent for proactive health | Checks queue depth, worker liveness, systemd health, disk, and DB size. Silent when healthy; opens human_tasks only when something needs attention. Observe-only — no code, PRs, or commits. |
| Proposal routing lives in one shared module | The dispatcher and TUI would otherwise duplicate (and diverge on) maintainer-proposal routing. shared/agent_routing.ts is the single source of truth, consumed via TypeScript project references and #shared/* subpath imports. |
| Agent repo scoping is enforced at dispatch | Single-repo agents (builder, memoriser, documenter, maintainer, auditor) are locked to the harness repo; repo-agnostic agents (developer, reviewer) use the product-repo wildcard. The routing.yaml agents section is binding, so harness tasks are never dispatched to the wrong repo. |
A read-only human_task_check.sh hook | Agents need to query human_task state without ad-hoc sqlite3 queries that break when the schema evolves. The hook gives a controlled, read-only JSONL interface. |
| Auto-approval for low-risk maintainer proposals | Docs-only or minor-memory proposals are auto-approved by the dispatcher, reducing operator toil while keeping code changes operator-gated. |
| The maintainer persists acceptance-rate metrics for self-calibration | Cumulative approval/decline totals by target agent and priority band are persisted and read before drafting proposals. Low acceptance rates are an advisory calibration signal, closing the loop between proposal generation and operator preference. |
| The auditor is an infrastructure-security agent for the harness | It scans the harness for command injection, hardcoded credentials, unsafe file ops, path traversal, sudo patterns, and network exposure — infrastructure, not application dependencies — on a weekly timer with its own checklist. |
| Defense-in-depth sudo block for agents | pre_tool_gate.sh blocks all sudo by agents beyond the script-name gate, with an explicit builder exception for apply-local-by-agent.sh. |
| Long-term direction: a self-improving agent ecosystem | The system continuously improves itself through the maintainer→builder loop, self-documents, refactors its memory, and grows agent capabilities — evaluating proposals against the operator-defined roadmap and executing within guardrails (secrets, auth, and destructive ops stay operator-gated). New agent types are deferred until existing workflows are bulletproof (stability first). |
Data
| Decision | Rationale |
|---|---|
source_meta_json on mas_tasks for source back-channel data | Keeps the schema generic; source-specific fields (e.g. Linear ids/urls) don't pollute top-level columns. |
session_id captured from the stream-json init event, stored on running_workers | Required for --resume. Captured async from stdout so it doesn't block worker start. |
plan_json validated by Zod in the dispatcher, not the delegator | The delegator is an agent; validation belongs in the TypeScript service that acts on the plan. |
schema_version: "1" literal on every plan | Forward compatibility — the dispatcher can branch on version when the schema evolves. |
| Scheduled agents write per-session artifacts | Each writes a markdown summary to agents/<agent>/artifacts/<plan_id>.md, surfaced in the TUI Artifacts tab, so the operator sees what a run did without reading raw transcripts. |
The routing.yaml scheduled section is the canonical reference | Every entry carries prompt and repo; the TUI reads them for manual triggers and run-scheduled-agent.sh passes them to the dispatcher. The gc timer's pipeline: [] is intentional — infrastructure-only, no agent. |
Ops
| Decision | Rationale |
|---|---|
The listener binds 0.0.0.0 in systemd but 127.0.0.1 in code | Caddy sits in front on the VM, so binding all interfaces behind it is acceptable; local dev stays loopback-only. |
The delegator is spawned detached with child.unref() | The listener returns 202 immediately regardless of delegator duration, decoupling HTTP latency from agent runtime. |
| Worker stdout/stderr is written to a JSONL session file, not piped to the dispatcher | Avoids backpressure blocking the dispatcher; session files are inspectable post-mortem via the TUI. |
bootstrap-vm.sh runs on every deploy (idempotent) | Schema migrations and path creation stay in sync with code without a separate migration tool; it also prunes stale git worktree metadata. |
| Unified workspace handling — worktrees for the harness, clones for product repos | gc-workspaces.sh uses git worktree remove for harness worktrees and rm -rf for product-repo clones, detected in a single loop. |
ccr.service is not restarted during deploys | Restarting ccr kills in-flight LLM connections. It is a routing proxy that doesn't need restarting on code changes; only the listener and dispatcher restart. |
| Transient ccr proxy errors are retried at multiple layers | A synthetic model error ("Content block is not a text block") is treated as retryable close to where it occurs — in-process worker retry, a listener dedup-bypass retry, and a stop-hook retry — so a transient proxy fault never fails a task. |
Deferred
Items deliberately not decided yet.
| Item | Trigger to revisit |
|---|---|
| Memory log rotation (daily → weekly → monthly) | Session-file storage exceeds 1 GB |
depends_on in the plan schema (sequential workers) | First multi-stage pipeline need |
| Per-user identity (CLAUDE.md per human) | Second human operator onboarded |
| Model pinning per agent | Observed quality/cost divergence in session logs |
| Human_task SLA / escalation | An open task sits unresolved > 24h |
| In-flight plan recovery after VM snapshot restore | First data-loss incident |