PRs #56/#57/#58 — v0.9.3: synlynk Goes Always-On

The Broader Goal at the End of the Previous PR

Post 21 ended with a clear statement of what had to come next:

"Team features at v0.9.2 complete the synchronous layer of the multi-agent OS. The next layer is the async daemon (v0.9.3): synlynk daemon start registers a background process that monitors project-docs/, fires context regeneration on writes, and exposes a local HTTP endpoint for agents to pull context without invoking the CLI. This moves synlynk from an on-demand tool to an always-running substrate — the step before the relay tier."

That goalpost was precise. The daemon had been on the roadmap since v0.8. The question going into this work wasn't what to build but how correctly to build it — the failure modes in a background daemon are nastier than anything in the synchronous path.

---

Strategic Shifts in This PR

Two things changed between that goalpost and what shipped.

First: a three-agent delivery protocol, fully exercised for the first time. v0.9.3 was the first release where task allocation across Claude, Agy, and Codex was enforced end-to-end with a formal rule: PRs reviewed by non-authoring agents; fixes applied only by the authoring agent. Every previous release had humans doing this coordination informally. For v0.9.3 it was a strict protocol:

Tasks	Author	Reviewer
1–3: schema, reconcile/dispatch, SynlynkDaemon + HTTP API	Claude	Agy
4: synlynk daemon CLI wiring	Agy	Codex
5–6: --install-service/--uninstall-service + VERSION bump	Codex	Claude

The authoring-agent-fixes-only rule was tested: Codex twice failed to apply a log path correction (~/synlynk/ vs ~/.synlynk/), and both times Claude (as reviewer, not author) had to re-dispatch rather than patch it directly. The second time, after Codex again baked in the wrong path with a regression test locking it, Claude applied the fix directly as a reviewer correction — the rule's edge case.

Second: correctness properties discovered during design review became the most important part of the implementation. Three bugs existed in the plan's pseudocode before a single line was written. Agy's review of PR #56 caught all three. Each one would have been silent in production.

---

What This PR Shipped

Daemon architecture

SynlynkDaemon subclasses WatchDaemon — it inherits the double-fork daemonization, pidfile management, mtime polling on project-docs/, and context.md regeneration. It adds exactly two things: a second thread running an HTTP server on localhost:27471, and _dispatch_ready_jobs() called on every poll tick.

The separation was deliberate. synlynk watch still works unchanged. The daemon is a superset, not a replacement — users who only want background context regeneration don't need the HTTP server or job queue.

class SynlynkDaemon(WatchDaemon):
    HTTP_PORT = 27471

    def _run_loop(self):
        # Start HTTP thread
        http_server = _ReuseAddrHTTPServer(("127.0.0.1", self.HTTP_PORT), handler_class)
        threading.Thread(target=http_server.serve_forever, daemon=True).start()

        # Poll loop (inherited cadence + reconcile/dispatch injected each tick)
        while True:
            time.sleep(interval)
            # ... mtime check → generate_context() if changed ...
            _reconcile_daemon_jobs()
            _dispatch_ready_jobs(max_parallel=max_parallel)

The daemon_jobs table

CREATE TABLE IF NOT EXISTS daemon_jobs (
    job_id       TEXT PRIMARY KEY,
    agent        TEXT NOT NULL,
    task         TEXT NOT NULL,
    status       TEXT NOT NULL DEFAULT 'queued',   -- queued → running → done | failed
    priority     INTEGER NOT NULL DEFAULT 5,        -- 1 (highest) to 10 (lowest)
    depends_on   TEXT NOT NULL DEFAULT '[]',        -- JSON array of job_ids
    pid          INTEGER,
    enqueued_at  TEXT NOT NULL,
    started_at   TEXT,
    completed_at TEXT,
    exit_code    INTEGER,
    log_path     TEXT
);

Three correctness properties were the critical work in this table's dispatch logic:

Zombie-safe reaping. os.kill(pid, 0) returns True for zombie processes. A child process that exits while the daemon holds no reference to its Popen object stays in the process table as a zombie; os.kill(pid, 0) succeeds, the daemon thinks it's still running, and the job never transitions out of running. The tests used mock PIDs (99999999) that raise ProcessLookupError immediately, so they couldn't catch this. Fixed with os.waitpid(pid, os.WNOHANG), with ChildProcessError fallback for processes adopted by init after a restart.

Dependency failure propagation. The original dispatch checked WHERE status='done' for all dependencies. If a dependency failed, its downstream job stayed queued forever — a silent deadlock. Fixed by checking dependency statuses explicitly: if any dep is failed, the blocked job is immediately marked failed rather than waiting.

Per-job commit. The original loop batched all spawned jobs in a single conn.commit() at the end. A daemon crash between spawning a process and committing left the job as queued in the DB while the process was already running — producing a duplicate execution on restart. Fixed by committing immediately after each spawn.

HTTP API (10 endpoints, stdlib only)

Method	Path	Purpose
GET	/context	context.md as {"content": "..."} or plain text
GET	/status	{running, uptime_s, pid, jobs: {queued, running, done, failed}}
GET	/jobs	All jobs; ?status= filter
GET	/jobs/<id>	Single job + last 100 log lines
POST	/dispatch	Enqueue job: {agent, task, story_id?, priority?, depends_on?}
GET	/stories	Story list from state.db
GET	/stories/<id>	Single story
GET	/capability	Capability ratings by agent × domain
GET	/sentinel	Parsed sentinel.md alert objects
POST	/checkpoint	Force-regenerate context.md immediately

Two implementation details that mattered: _ReuseAddrHTTPServer (allow_reuse_address = True) prevents Address already in use errors on rapid daemon stop/start. A threading.Lock on SynlynkDaemon guards concurrent generate_context() calls — the HTTP /checkpoint handler and the poll thread's on_change() path both write to context.md, and without the lock a concurrent call corrupts the file.

Service installation

synlynk daemon --install-service

Platform detection:

1. sys.platform == 'darwin' → launchd plist at ~/Library/LaunchAgents/com.synlynk.daemon.plist, launchctl load -w
2. shutil.which('systemctl') → systemd user unit at ~/.config/systemd/user/synlynk-daemon.service, systemctl --user enable --now
3. Fallback → @reboot synlynk daemon start appended to crontab

--uninstall-service reverses each path. FileNotFoundError (service not installed) is caught and prints gracefully.

CLI surface (Task 4, authored by Agy)

synlynk daemon start               # fork-daemonize
synlynk daemon stop                # SIGTERM to pidfile pid
synlynk daemon status              # running state + queue depth + HTTP URL
synlynk daemon restart             # stop then start
synlynk daemon --install-service
synlynk daemon --uninstall-service

The daemon subparser was wired into main() with stub functions _daemon_install_service() and _daemon_uninstall_service() that Codex later replaced with real implementations in Task 5.

Test suite

432 tests on merge. New tests added across all three PRs:

• Task 1: daemon_jobs table exists and accepts inserts
• Task 2: zombie reaping via os.waitpid, dep-failure propagation, max-parallel cap, per-job commit ordering
• Task 3: 16 tests covering all 10 HTTP endpoints + 3 daemon lifecycle methods
• Task 4: 6 CLI integration tests (status, stop, default no-action, restart sequence, both service stubs)
• Task 5+6: platform-specific install/uninstall tests (mocked subprocess.run), VERSION == '0.9.3'

The restart test (test_daemon_cli_restart_not_running) went through two iterations: Agy's first version only asserted "not running" in stdout, which would pass even if restart called stop() but not start(). Codex's review caught the gap. Agy's second version uses a call tracker:

calls = []
monkeypatch.setattr(SynlynkDaemon, 'stop', lambda self: calls.append('stop') or print('  ✦ daemon not running'))
monkeypatch.setattr(SynlynkDaemon, 'start', lambda self: calls.append('start'))
# ...
assert calls == ['stop', 'start']

---

Brainstorm Visuals Used

The brainstorm session produced two visual companion pages that informed the design:

• docs/brainstorm/v0.9.3-async-daemon/daemon-approaches.html — the three architecture options (extend WatchDaemon vs new SynlynkDaemon subclass vs standalone HTTP process). Architecture B (new subclass) was chosen because it reuses the double-fork + pidfile + mtime polling from WatchDaemon without entangling synlynk watch.
• docs/brainstorm/v0.9.3-async-daemon/daemon-architecture.html — the two-thread process model, all 10 HTTP endpoints, job queue dispatch rules, and service registration options side-by-side.

---

What This Achieved on the Path to Autonomy

Agents can now operate without a human invoking the CLI.

Before v0.9.3, every agent interaction required synlynk exec — a human (or human-scheduled job) invoked the CLI, which injected context, ran the agent, and recorded telemetry. The agent couldn't self-service context or queue follow-on work without another CLI invocation.

After v0.9.3:

• Any process on localhost can GET localhost:27471/context and get the current project state without invoking synlynk
• Any process can POST localhost:27471/dispatch to enqueue work for a specific agent with priority and dependency ordering
• The daemon keeps context current automatically — no poll-and-refresh loop needed
• Jobs survive daemon restarts and won't duplicate on crash

The three-agent delivery protocol also advanced the autonomy goal in a different way: for the first time, the authoring/reviewing/fixing roles were enforced mechanically across a full feature. The protocol surfaced 7 distinct issues that would have shipped as bugs — none were caught by CI. That's the capability engine building its own case: agents reviewing each other's work is not a courtesy, it's a correctness mechanism.

---

Strategic Note: The Goal at the End of This PR

v0.9.3 establishes synlynk as an always-running process. The HTTP API is exposed on localhost:27471 — reachable from any process on the same machine.

The next step is making it reachable from other machines in the same workgroup.

v0.9.4: Workgroup Relay — a WSS/443 relay that lets agents on different machines share context and dispatch across a workgroup without a shared filesystem. Three deployment modes: LAN (no internet dependency), Cloudflare Tunnel (zero-config public relay), VPS (self-hosted). The daemon's HTTP API becomes the data source; the relay becomes the transport.

Once the relay ships, a developer on one machine can dispatch work to an agent running on another machine's daemon, with context propagating automatically. That's the workgroup tier. The OS tier above that is what v1.0 builds toward.