PR #772 — jobs reap: Kill Zombie `running` Rows for Real
jobs reap — Kill Zombie running Rows for Real
Broader goal (end of previous PR)
PR #762 windowed sentinel_crit so lifetime sentinel.md history no longer kept platform ops RED. The scoreboard could see recent failures honestly — and immediately showed the next lie: hundreds of daemon_jobs rows stuck at status=running with dead PIDs.
What moved the goalpost
A same-day ops triage reaped 482 zombies by hand (45 in the three main project DBs). STALL_NO_OUTPUT and HARNESS_INTERNAL_TIMEOUT killed processes and wrote CRITICAL sentinels but often left the row running. _reconcile_daemon_jobs also left dead PIDs as open-ended unknown (or skipped null PIDs). Manual SQL is not a product.
What this PR ships
synlynk jobs reap— dry-run by default;--applymarks zombiestimed_out/exit_code=-9;--all-projectswalks every~/.synlynk/projects/*/state.db.- Auto-reap on sentinel write — when
_write_sentinel_alertemitsSTALL_NO_OUTPUTorHARNESS_INTERNAL_TIMEOUT, parsejob-*and flip still-running daemon rows immediately. - Hardened
_reconcile_daemon_jobs— null PID and dead PID (including non-child processes) becometimed_outinstead of lingeringrunning/ vagueunknown. - Helpers:
_pid_is_alive,mark_daemon_job_terminal,scan_zombie_running_jobs,apply_reap_zombies,auto_reap_job_from_sentinel. - Tests for dry-run/apply, auto-reap via sentinel, CLI parser, and reconcile expectation update.
On the long arc
Autonomous multi-agent dispatch needs a job table that matches process reality. Windowed sentinels (#751) fixed the alert log; jobs reap (#753) fixes the queue truth that platform ops L1 rates depend on.
New goalpost
- Operators have a one-command hygiene path (
jobs reap --all-projects --apply). - Future STALL/TIMEOUT cannot recreate the 482-zombie pile without an auto flip.
- Still open: agy timeout root cause (#750), cost capture gaps (#752), full #701 epic.