Quality Signal Architecture

Three tiers feeding one ledger row — automated baseline, verifier primary, human override

AUTO Objective signals — always captured, zero human cost
signal_source = 'auto' · weight: 0.60
test_pass_rate
% tests passing after job completes. Scraped from CI log or test runner output in job log.
build_success
Boolean. Job exit code + build/lint output parsed from log.
rework_count
Number of re-dispatch cycles before story reached done. First-pass success vs trial-and-error.
duration_vs_estimate
Actual token/time cost vs story.estimated_tokens. Efficiency signal.
pr_review_cycles
How many PR review rounds before merge. More cycles = lower quality signal.
verified_by_ci
CI pipeline outcome after PR merge. Ground truth for correctness.
Auto-score computed as weighted composite → normalised to 0–10. Stored as quality_auto. Used when no verifier or human rating exists. Flagged low-confidence in Tokq.
VERIFIER Peer agent rating — primary quality score in trio pipeline
signal_source = 'verifier' · weight: 0.85

At the end of a trio pipeline run, the Verifier agent is dispatched with the Builder's diff, the story's done_criteria, and the auto-signals. It emits a structured review ending with:

# synlynk-meta
quality=8
correct=true
rework_needed=false
verifier_model=gemini-2.5-pro

Verifier's own model_version is recorded separately — the rating is a function of both the Builder's model and the Verifier's model. Important for Tokq: a high verifier score from a capable verifier is worth more.

HUMAN Override — highest weight, not required
signal_source = 'human' · weight: 1.0
synlynk score add <story-id> 9 [--note "nailed it"]
synlynk score add <story-id> 3 --rework --note "missed edge cases"

Human rating overwrites auto + verifier in the routing score. Carries a note field that flows into Tokq Memory Unit description.

Effective score resolution — what gets written to capability_ratings
Human rating
if present → wins
→
Verifier rating
else if present
→
Auto composite
low-confidence flag
+
Auto signals
always appended as metadata

Auto signals are always stored as metadata columns regardless of which tier provides the primary quality score. Routing uses effective score; Tokq exposes all tiers for subscribers to make their own judgments.