PR #29 — v0.3.1: When the OS Learns to Self-Diagnose

PR: feat: synlynk v0.3.1 — sentinel + observability hardening
Branch: feat/v0.3.1-sentinel-observability
Status: open
Contents: 9 new features, 40 new tests, 12 commits — all in bin/synlynk.py, zero new dependencies

---

The Broader Goal at the End of PR #28

The 2026-06-07 design session locked the architecture from v0.5.0 through v1.0: state.db, agent identity, workspace model, dispatch modes, entitlements. The flat-file era was declared complete with v0.4.0. The immediate next step, per PR #28, was implementation of v0.4.0 — the Trio Protocol (Architect → Build → Verify pipeline, shared conventions, synlynk run).

The roadmap read: v0.4.0 is immediately ready, no gaps, all design decisions locked.

---

The Strategic Shift: One Prerequisite Hidden in the Schema

Before writing the v0.4.0 implementation plan, we upgraded the locally-installed synlynk from v1.2.0-lite to v0.3.0. That's when the regression surfaced.

v0.3.0 switched exec_command to TTY pass-through (subprocess.Popen with inherited stdio) to support interactive CLIs like Claude Code and Gemini. The right call for interactive mode — but it silently dropped extract_tokens() and update_costs() in the process. Token scraping had existed since v1.2.x. It was gone.

Then we looked at the v0.5.0 state.db schema spec:

agent_quotas:
  used_tokens: INTEGER  -- populated by extract_tokens(), unchanged

extract_tokens() was listed as a prerequisite of v0.5.0 — explicitly, in the spec. A regression that blocks two releases upstream is not a cosmetic fix. It's a pre-condition.

The pivot: Do v0.3.1 before v0.4.0. Restore the regression. While there, harden the sentinel and observability layer — a cohesive set of health signals that the OS should have had from the start.

---

What This PR Shipped

The Foundation: Structured Sentinel Alerts

Everything else builds on two new functions:

def _write_sentinel_alert(severity: str, code: str, message: str) -> None:
    # Appends to .synlynk/sentinel.md:
    # - [CRITICAL] [2026-06-10 14:23] ZOMBIE_DAEMON: message

def _read_sentinel_alerts(severity: Optional[str] = None) -> list:
    # Returns alert lines, optionally filtered by severity
    # Old free-form lines (pre-v0.3.1 format) preserved as-is

The severity filter uses a field-level regex (re.match(r'^- \[([A-Z]+)\]', line)) rather than a substring check — so [WARN] in the message body doesn't false-positive on a CRITICAL filter. A subtle distinction that matters when the sentinel layer is the last line of defense.

Restored: extract_tokens + update_costs

extract_tokens(output_text: str) -> tuple matches five patterns across all supported CLIs:

Pattern	CLI
Input tokens: N / Output tokens: N	claude text mode
"input_tokens": N, "output_tokens": N	claude JSON mode
Tokens used: N input, N output	gemini
prompt_tokens: N / completion_tokens: N	openai-compat
Total tokens: N (80/20 split fallback)	any

update_costs appends a markdown row to project-docs/costs.md at $0.003/1K input + $0.015/1K output.

Non-Interactive Token Scraping (the Hard Part)

The reason extract_tokens was dropped was real: you cannot capture stdout from a process that's also using it as a TTY. The solution is detection + conditional capture:

def _is_interactive(cmd_args: list) -> bool:
    NON_INTERACTIVE = ["--no-tty", "--output-format json", "--print",
                       "--non-interactive", "-p "]
    return not any(flag in " ".join(cmd_args) for flag in NON_INTERACTIVE)

For non-interactive execs, stdout is captured via a tee thread that streams to the terminal in real time while buffering for token scraping:

def _tee_process(process, buffer: list) -> None:
    for line in iter(process.stdout.readline, b''):
        sys.stdout.buffer.write(line)
        sys.stdout.buffer.flush()
        buffer.append(line.decode('utf-8', errors='replace'))
    process.stdout.close()

Interactive mode (TTY pass-through) is completely unchanged. The tee path is only taken when _is_interactive() returns False.

After each non-interactive exec with in_tokens > 0, a cost pulse is printed:

  ⚡ Tokens: 4,821 in / 312 out  |  est. $0.0191

Zombie Daemon Detection

WatchDaemon._is_running() (a os.kill(pid, 0) check) was extended into _health() returning a tri-state:

def _health(self) -> str:  # "running" | "stopped" | "zombie"

"zombie" = pidfile exists, process dead. check_daemon_health() writes a CRITICAL sentinel on zombie and is called from cmd_status(). Before this, a dead daemon was silent.

Stall Detection

set_state("active") writes .synlynk/state at exec start. If state is still "active" and the file's mtime is older than exec_timeout_minutes (default 30, configurable), the exec has likely stalled:

def check_stall() -> None:
    age_minutes = (time.time() - os.path.getmtime(state_file)) / 60
    timeout = load_config().get("exec_timeout_minutes", 30)
    if age_minutes > timeout:
        _write_sentinel_alert("WARN", "STALL", ...)

Detection is retrospective (runs on next status or exec call) — no background thread required. Single-file CLI constraint preserved.

check_sentinel_patterns: Three Detectors in One

The old check_flatline() was replaced and extended into check_sentinel_patterns():

Pattern 1 — Flatline: Last 3 telemetry exec events have the same command and all non-zero exit codes → CRITICAL. The original flatline detector, now properly wired.

Pattern 2 — Success loop: Last 5 exec events same command, all exit 0, within 10 minutes → WARN. An automated script calling synlynk exec claude in a tight loop burns tokens with no progress — this catches it before the budget is gone.

Pattern 3 — Quota-exhausted: Known phrases in captured output (case-insensitive, first match, break):

QUOTA_PATTERNS = [
    "rate limit", "quota exceeded", "resource exhausted",
    "billing", "insufficient_quota", "too many requests",
    "RESOURCE_EXHAUSTED",
]

→ CRITICAL. One exec hitting quota and failing 3 times previously took 3 failures to get a FLATLINE alert. Now it's caught immediately on the first quota hit.

Pre-Exec Gate

Critical alerts now block exec:

def _check_pre_exec_gate(force: bool = False) -> bool:
    criticals = _read_sentinel_alerts(severity="CRITICAL")
    if criticals and not force:
        print("  Exec blocked by CRITICAL sentinel alert.")
        return False
    return True

WARN alerts print but do not block. synlynk exec --force bypasses CRITICAL gates for emergency use.

Context Bloat Warning

generate_context() now checks the size of .synlynk/context.md after writing it:

• > 32 KB → mild warning
• > 64 KB → strong warning with archiving suggestion

context.md is injected into every exec. At 64 KB that's thousands of tokens on every run. The warning is informational only — never a blocker.

Burn Rate + Runway

synlynk status now shows:

  Budget: $1.24 / $10.00  (12.4%)  ·  23 requests
  Burn:   $0.019/exec avg  |  ~461 execs remaining at current pace

_compute_burn_rate() reads the last 10 costed telemetry events (those with in_tokens > 0), computes avg $/exec, and divides remaining budget. Returns (0.0, None) when fewer than 3 costed events — not enough data to show a meaningful rate.

synlynk sentinel CLI

synlynk sentinel list          # List all active alerts
synlynk sentinel clear          # Clear all structured alerts
synlynk sentinel clear --severity WARN  # Clear only WARN
synlynk sentinel clear --code STALL     # Clear by alert code

Old free-form sentinel lines (pre-v0.3.1 format) are preserved by sentinel_clear — it only removes structured [SEVERITY] [TIMESTAMP] CODE: format lines.

---

Implementation Approach: Subagent-Driven TDD

This was the first synlynk PR implemented entirely via subagent-driven development: a fresh Claude subagent per task, dispatched with a self-contained prompt containing the full task text, expected test output, and exact code to write. Two review subagents followed each implementer — one for spec compliance, one for code quality.

The process caught two real bugs before they reached the PR:

1. The severity filter used f"[{severity}]" in line — a substring match that would false-positive if the message body contained the severity string. Fixed to re.match(r'^- \[([A-Z]+)\]', line).
2. check_flatline() was left as dead code when the stub was replaced. Removed; the 4 existing tests were updated to call check_sentinel_patterns with "type": "exec" in their telemetry fixtures.

10 tasks, 12 commits, 40 new tests. 123 total, all passing.

---

What This Achieved on the Path to Autonomy

An OS that cannot self-diagnose will fail silently and expensively. Before v0.3.1, synlynk could detect a flatline — 3 consecutive failures — but nothing else. A zombie daemon, a stalled exec, a quota exhaustion, a success loop burning tokens in a tight cycle: all invisible.

v0.3.1 closes that gap. synlynk status is now a health dashboard:

• Daemon alive or zombie?
• Last exec complete or stalled?
• Quota intact or exhausted?
• Burn rate sustainable or accelerating?
• Context size reasonable or bloated?

These are the signals an autonomous system needs to decide whether to continue or escalate to the human. Without them, the human is always in the loop by necessity — the only entity who can tell if something is wrong by looking at the terminal. With them, synlynk status can make that call first.

The pre-exec gate is the first step toward a sentinel layer that blocks bad execs before they happen. v0.5.0 introduces hard token ceilings at the story level. But the CRITICAL → block pattern established here is the same pattern that ceiling enforcement will use.

And extract_tokens() is back — feeding real cost data into project-docs/costs.md and into the telemetry events that v0.5.0's agent_quotas.used_tokens will consume.

---

The New Goalpost

v0.4.0 — Conventions + Trio Bootstrap — is the next release. The Architect → Build → Verify pipeline, shared .rules/ conventions, and synlynk run <task> as the first autonomous task primitive. All design decisions are locked. The implementation plan is next.

The flat-file era ends at v0.4.0. v0.3.1 hardened the OS layer that ships with it.