MC orchestration scheduled-task simplification review — 2026-05-28

Verdict

Alive, but over-layered. The MC orchestration scheduled tasks are mostly firing and the core path is functioning, but the control plane has accumulated overlapping minute loops, watchdogs, cleanup jobs, and meta-tuning tasks. The simplification target should be: one minute-level intake/dispatch loop, one 15-minute operator loop, one external Hermes watchdog, and a small set of true cleanup/service monitors.

Evidence gathered read-only:

• ticket-pickup, needs-input-pickup, triage-untriaged, persistent-luci-watchdog, queue-reaper, reap-zombie-workers, cron-watchdog, scheduler-watchdog, systemd-watchdog, and luci-operator all have fresh successful runs.
• luci-persistent tmux session exists and the watchdog reports it online.
• persistent-luci-branch-guard reports master.
• Hermes external cron job MC control-plane watchdog (quiet, 15m) is enabled and running every 15m; latest archived run was silent, while latest state still records one human-action issue: MC-3896 awaiting Elmar.
• Recent issue: pickup had repeated HTTP Error 409 dispatch warnings around MC-4323, now parked as blocked after repeated runtime-send failures. That is not a scheduler-wide outage, but it proves the system still sees dispatch/runtimes churn.

What each task does and simplification verdict

External Hermes cron

• MC control-plane watchdog (quiet, 15m)
• Layer: Hermes cron, outside MC scheduler.
• Does: probes MC API/service state, core task freshness, stale in-progress/review, orchestrator inbox, human-action waits; writes latest report under ~/.hermes/cron/state; Telegrams only when non-empty output is needed.
• Evidence: enabled, schedule every 15m, last status ok, latest report says MC API 200, ccgram.service active, luci-dashboard.service active, core tasks fresh, one human-action wait (MC-3896).
• Verdict: Keep. This is the right outside-the-scheduler sentinel. It should become the single external alert for MC orchestration health.

Operator / control-plane decision loop

• luci-operator
• Schedule: every 15m, background.
• Does: broad high-agency operator pass over active tickets, needs-input, done/review quality, scheduler task runs, runtime sessions, disk/worktree pressure, tmux/service state, control-plane health. Can heal safe drift and create/operator tickets.
• Evidence: fresh successful run; one failure in 72h caused by transient missing script path during repo state, followed by many successes.

• Verdict: Keep, but narrow the role. It should be the 15-minute decision-maker/gatekeeper, not also another dispatcher/reaper/scheduler watchdog. Treat other jobs as sensors/actuators feeding it.

• luci-operator-tuner

• Schedule: nightly.
• Does: reads logs/luci_operator.jsonl, scores operator behavior, writes a report, files an MC ticket, and can ask a Claude subagent for patch suggestions.
• Evidence: fresh successful run filed MC-4333 / Operator tune — 2026-05-28, assigned to Elmar.
• Verdict: Simplify aggressively. This is meta-noise if it creates daily Elmar-facing tickets. Make it weekly or anomaly-triggered, and route to Luci unless it needs a human product decision.

Intake / dispatch

• ticket-pickup
• Schedule: every minute.
• Does: normal dispatch loop. Also reaps idle terminal-state tmux sessions, routes stale needs_input and in_review to orchestrator inbox, drains orchestrator inbox through luci-persistent, triages inbox mentions, dispatches Luci tickets, and dispatches Larry tickets.
• Evidence: fresh successful run. Recent warnings showed HTTP Error 409 dispatch failures for MC-4323 and related tickets; latest run had no tickets at that moment.

• Verdict: Keep as the single minute-level intake/dispatch loop. It already does most of the fast-path work.

• needs-input-pickup

• Schedule: every minute.
• Does: a fast lane for tickets in needs_input. Code comments say this was fixed to be “truly needs-input-only” under the same single-flight pickup lock, avoiding the old double-dispatch race.
• Evidence: fresh successful runs, but recent warning output still involved dispatch conflict symptoms. It is mostly idle and overlaps with the normal minute-level ticket-pickup cadence.

• Verdict: Candidate to retire or convert to event-driven. Its original body says normal pickup is every 15m, but normal pickup now runs every minute. If ticket-pickup can handle requeued replies fast enough, this task is redundant.

• triage-untriaged

• Schedule: every minute.
• Does: direct Gemini triage for unassigned inbox/todo tickets; assigns high-confidence tickets.
• Evidence: fresh successful run, usually processed 0 ticket(s).
• Verdict: Keep the capability but reduce/merge the schedule. Either fold it into the single minute intake loop or run every 5 minutes / event-driven. Running a dedicated minute task that usually processes zero is unnecessary surface area.

Reapers / cleanup

• queue-reaper
• Schedule: every 15m.
• Does: expires stale pending queued messages, reclaims expired leases, fails exhausted retries, cleans old records.
• Evidence: fresh successful run: 0 expired, 0 reclaimed, 0 failed, 0 cleaned.

• Verdict: Keep. It owns a distinct queue-table concern.

• reap-zombie-workers

• Schedule: every 10m.
• Does: kills tmux mc-MC-* sessions whose tickets are terminal (done/cancelled) after grace period. It is a safety net for inline cleanup.
• Evidence: fresh successful run scanned 4 sessions, killed 0.
• Overlap: ticket-pickup already calls reap_terminal_state_tmux_sessions() every minute for sessions whose ticket left todo/in_progress.

• Verdict: Candidate to merge/reduce. Keep as a belt-and-braces independent fallback only if pickup/runtimes remain fragile; otherwise reduce to hourly or retire after pickup reaping is trusted.

• worktree-reaper

• Schedule: weekly.
• Does: broad clean worktree pruning across workspace and PKA, only when safe; avoids active/open tickets.
• Evidence: last weekly run completed.

• Verdict: Keep. Distinct low-frequency maintenance.

• mc-orchestrator-inbox-cleanup

• Schedule: daily.
• Does: direct SQL marks stale low-priority scheduler rows in orchestrator_inbox as processed after 24h. Body says insert-time auto-expiry now exists, so this is belt-and-braces legacy cleanup.
• Evidence: fresh successful run.
• Verdict: Candidate to retire or run weekly/monthly. Direct SQL cleanup is easy to forget and “processed” must not mean “acted.” If insert-time auto-expiry is reliable, this should go away.

Persistent Luci

• persistent-luci-watchdog
• Schedule: every 5m.
• Does: checks luci-persistent liveness, restarts after 2 consecutive offline observations, and manages context compaction/clear ladder.
• Evidence: latest output: luci-persistent OK, online, tokens below thresholds.

• Verdict: Keep. This is core to the orchestrator control surface.

• persistent-luci-branch-guard

• Schedule: every 15m.
• Does: checks persistent Luci repo branch is master; alerts on drift, does not autoswitch.
• Evidence: latest output: branch-guard: ok (master).

• Verdict: Merge into persistent-luci-watchdog. It is tiny and related to the same persistent-session health domain.

• rotate-luci-session

• Schedule: daily 03:00.
• Does: clears pinned Claude session id and kills process so the wrapper respawns fresh without huge resumed transcript.
• Evidence: latest run completed.
• Verdict: Keep. It solves a distinct context-growth failure mode. Keep separate because it is intentionally disruptive and scheduled at a quiet time.

Watchdogs / sweeps

• orchestrator-board-sweep
• Schedule: hourly.
• Does: auto-closes recovered watchdog tickets; flags stale in_review and needs_input into orchestrator inbox; releases waiting tickets when blockers recovered.
• Evidence: fresh successful run.
• Overlap: luci-operator also audits stale ticket state; ticket-pickup also triages stale needs_input/in_review into inbox.

• Verdict: Candidate to fold into luci-operator. If the operator is the 15-minute board decision-maker, hourly board sweep can become a subroutine rather than a separate task.

• scheduler-watchdog

• Schedule: hourly.
• Does: creates deduped MC tickets for YAML parse errors, missed fires, execution failures, alert-channel health, disabled critical tasks, and completed dispatch anomalies.
• Evidence: fresh successful run with empty output.
• Overlap: Hermes external watchdog and cron-watchdog also monitor task freshness/overdue state.

• Verdict: Keep, but make it the only MC-internal scheduler-ticket creator. It is useful because it creates durable MC tickets; avoid duplicate Telegram alerts from other scheduler monitors.

• cron-watchdog

• Schedule: every 30m.
• Does: calls MC /api/v1/scheduler/inspect and Telegram-alerts on overdue tasks.
• Evidence: latest inspected 140 tasks, 0 overdue.
• Overlap: scheduler-watchdog checks missed fires/failures and creates MC tickets; Hermes external watchdog checks core task freshness outside the scheduler.

• Verdict: Candidate to retire. A scheduler task watching the scheduler is weaker than Hermes external cron, and it duplicates scheduler-watchdog. Keep only if it covers non-core overdue alerts not covered elsewhere.

• systemd-watchdog

• Schedule: hourly.
• Does: alerts on systemd units transitioning to failed/recovered; includes persistent Luci state.
• Evidence: latest output: all watched services OK, 0 failed units.

• Verdict: Keep. Distinct service-layer monitor. Could feed its results to the external Hermes watchdog eventually, but not urgent.

• pickup-watchdog

• State: disabled/retired.
• Does/would do: re-enable suspended pickup tasks and alert.
• Evidence: no task_runs; frontmatter explicitly says enabled: false, disabled_reason: retired.
• Verdict: Remove from active mental model. Leave archived or delete after confirming no docs still refer to it as live.

Main overlaps

1. Three every-minute intake loops: ticket-pickup, needs-input-pickup, triage-untriaged.
2. Multiple stale-review/needs-input triagers: ticket-pickup, orchestrator-board-sweep, and luci-operator.
3. Duplicate tmux cleanup: ticket-pickup has reaping; reap-zombie-workers also reaps.
4. Duplicate scheduler monitors: Hermes external watchdog, scheduler-watchdog, and cron-watchdog all inspect task freshness/failures.
5. Persistent-session health split: persistent-luci-watchdog, persistent-luci-branch-guard, rotate-luci-session, plus systemd-watchdog all touch parts of persistent Luci health.
6. Meta-operator tuning creates human-facing noise: nightly luci-operator-tuner filed an Elmar ticket.

Recommended simplified shape

Keep as core layers

1. External sentinel
2. Keep: Hermes MC control-plane watchdog (quiet, 15m).

3. Purpose: outside-scheduler truth source; quiet on health; alert only genuine human/action issues.

4. Single minute intake/dispatch

5. Keep/expand: ticket-pickup.

6. Fold into it or trigger from it:

   • needs-input-pickup
   • triage-untriaged
   • lightweight stale needs_input/in_review routing
   • lightweight idle tmux reaping

7. Single 15-minute operator

8. Keep: luci-operator.
9. Purpose: durable decisions, gate/reject review, technical follow-up tickets, dependency cleanup, scheduler/runtimes summary.

10. Should not compete with the minute dispatcher.

11. Cleanup lane

12. Keep: queue-reaper, worktree-reaper.

13. Decide whether reap-zombie-workers remains independent or becomes less frequent.

14. Service/session lane

15. Keep: persistent-luci-watchdog, rotate-luci-session, systemd-watchdog.

16. Merge: persistent-luci-branch-guard into persistent watchdog.

17. Scheduler-ticket creator

18. Keep: scheduler-watchdog for durable MC tickets.
19. Retire/demote: cron-watchdog, unless it has unique overdue coverage.

Concrete simplification candidates, in order

1. Retire or event-trigger needs-input-pickup. Normal ticket-pickup already runs every minute and uses the same pickup lock.
2. Fold triage-untriaged into the single intake loop or slow it to every 5 minutes. It usually processes zero.
3. Merge persistent-luci-branch-guard into persistent-luci-watchdog. Same domain, tiny check.
4. Retire cron-watchdog after confirming Hermes external watchdog + scheduler-watchdog cover the same overdue conditions. Avoid duplicate scheduler monitors.
5. Fold orchestrator-board-sweep into luci-operator or keep only as a subroutine. Same board-hygiene domain.
6. Reduce reap-zombie-workers to hourly or make it emergency fallback only. Pickup already does minute-level session reaping.
7. Change luci-operator-tuner from nightly Elmar-facing tickets to weekly/anomaly-triggered Luci-owned reports. This cuts meta-noise.
8. Retire mc-orchestrator-inbox-cleanup if insert-time auto-expiry is confirmed. It is legacy direct SQL.

Suggested end-state task list

• ticket-pickup — every minute; single intake/dispatch/requeue/triage-light loop.
• luci-operator — every 15m; high-agency board/runtime/scheduler decision pass.
• Hermes MC control-plane watchdog — every 15m; outside-scheduler sentinel.
• scheduler-watchdog — hourly; durable MC tickets for scheduler faults.
• persistent-luci-watchdog — every 5m; liveness/context/branch health.
• rotate-luci-session — daily.
• queue-reaper — every 15m.
• worktree-reaper — weekly.
• systemd-watchdog — hourly.

Everything else should either be retired, merged, or downgraded to lower cadence.

Current “is it working?” answer

• Core MC orchestration: working, not clean.
• Persistent Luci: working now.
• Pickup: working, but recent runtime-send/409 churn shows fragility.
• Operator: working, but too broad and meta-tuner creates noise.
• Reapers: working, partly duplicated.
• Watchdogs: working, too many overlapping alert surfaces.
• Human action currently waiting: MC-3896 only in the external watchdog report.