MC-3718: Orchestrator Flow — Design Proposal

Date: 2026-05-19 · Ticket: MC-3718 · Status: APPROVED by orchestrator — decomposed into phase tickets. Key constraint: The 2026-04-20 tmux/worktree/xterm.js rearchitecture was REJECTED. This proposal is orthogonal — orchestrator routing, not browser display. No xterm.js, no per-ticket tmux, no new systemd service, no wholesale rebuild.

1. Current state — where the orchestrator IS and IS NOT

1.1 Human path (Elmar → MC composer → work)

The composer (templates/partials/session_composer.html) has three modes — Quick Ticket, Luci do it now, Chat with Luci — all hitting POST /api/v1/session_start.

• Quick Ticket → models.create_ticket() → mc.db (inbox/todo) → cron → mc_pickup.py → work_ticket() → claude subprocess. Orchestrator NOT involved.
• Luci do it now → create_ticket() status=in_progress → mc_pickup.py dispatch. Orchestrator NOT involved.
• Chat with Luci (plan) → _start_luci_chat_session() → mc_tmux.start_session("luci-chat-{slug}-{uuid}") — a NEW EPHEMERAL session, explicitly "not the persistent operator home session" (app.py:5468). Redirects to workbench. Dies with the conversation. No continuity.

1.2 Scheduled path (cron → task → outcome)

cron → scheduler.py tick → load_tasks() → run_task() → subprocess / direct-API / background. On success: silent. On failure: _task_notify() → Telegram direct. On 3× fail: create_mc_ticket(). The orchestrator never sees outcomes. Complete bypass.

1.3 What already exists and is close

luci-persistent tmux session + full API (persistent_luci.py — send(), harvest(), start/stop/switch; app.py:8336 endpoints) — the intended orchestrator, fully functional, but unwired as a front door. The infrastructure exists; the WIRING is missing.

2. Target architecture

luci-persistent becomes the single coordination layer. Both inbound paths route through it.

Human path (target): composer → POST /api/v1/persistent_luci/send → persistent session reasons (decompose/clarify/do/ticket/delegate) → response inline in composer, with cross-message continuity.

Scheduled path (target): scheduler.py task completes → outcome written to a new orchestrator_inbox table → mc_pickup.py drain cycle reads pending items → consolidated digest → persistent_luci.send() → orchestrator triages (routine=silent, repeated failure=ticket, auth=immediate Telegram preserved, actionable=Telegram+ticket).

Design options

• Option A — Orchestrator Inbox Queue (RECOMMENDED for scheduler): new orchestrator_inbox table; scheduler writes outcomes; mc_pickup.py drains, batches into a digest, sends to persistent session. Durable, batched, async, testable. ~1min latency — acceptable.
• Option B — Direct Persistent Route (RECOMMENDED for Phase 1 composer): composer hits persistent_luci/send directly. Simplest. Risk: concurrent sends serialise badly — fine for composer, not for scheduler at scale.
• Option C — MC Ticket as Orchestrator (fallback only): scheduled "orchestrator-review" task digests task_runs. High latency, board noise, doesn't fix the composer path. Fallback if P2 blocked.

Decision: Option B for Phase 1 (composer), Option A for Phases 2-3 (scheduler).

orchestrator_inbox schema:

CREATE TABLE orchestrator_inbox (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  source_type TEXT NOT NULL,      -- 'scheduler','pickup','api','self_heal'
  source_id TEXT,
  message TEXT NOT NULL,
  priority TEXT DEFAULT 'normal',  -- 'critical','normal','low'
  status TEXT DEFAULT 'pending',   -- 'pending','claimed','processed','ignored'
  created_at TEXT DEFAULT (datetime('now')),
  processed_at TEXT,
  claim_expires_at TEXT
);
CREATE INDEX idx_orchestrator_inbox ON orchestrator_inbox(status, priority, created_at);

3. Phased plan (each independently shippable)

Phase 1 — wire composer "Chat with Luci" to the persistent session

• session_composer.html: mode=plan submits to persistent_luci/send, polls harvest, shows response inline (heartbeat spinner; keep a "view in console" link to /console?session=luci-persistent).
• app.py: optional POST /api/v1/chat convenience wrapper (send+harvest); deprecate _start_luci_chat_session() / _luci_chat_prompt().
• Untouched: quick_ticket, implement_now, scheduler.py, mc_pickup.py.
• Risk: harvest 45s timeout — use polling UX, not one blocking call.

Phase 2 — scheduler posts outcomes to the orchestrator inbox

• Add orchestrator_inbox table migration (CREATE TABLE IF NOT EXISTS).
• scheduler.py: add _post_orchestrator_outcome(task, outcome, priority) (direct sqlite insert, non-fatal); call from run_task(). Auth failures STILL direct-Telegram immediately (parallel path). Keep existing _task_notify() during validation (run both).
• Inbox cleanup: DELETE ... WHERE status IN ('processed','ignored') AND created_at < datetime('now','-7 days').

Phase 3 — mc_pickup.py drains the inbox through the orchestrator

• mc_pickup.py: add drain_orchestrator_inbox(conn) — read ≤20 pending, format digest, atomic claim, persistent_luci.send(), poll harvest(), mark processed/ignored, rollback on error. Call in main() dispatch cycle after reap, before dispatch.
• Update persistent-session context: it receives inbox digests; surface critical, ticket actionable failures, silently ack routine.
• After 1 week stable: remove direct _task_notify() for non-critical/non-auth. Keep auth-failure direct Telegram.
• Graceful offline handling: session down → items stay pending, retry next cycle, WARNING not CRITICAL.

Phase 4 — orchestrator-initiated intent decomposition (OPTIONAL, deferrable)

Elmar states high-level intent; orchestrator decomposes into tickets/delegation. Start as advisor (responds, Elmar clicks to act), add autonomous action once trusted. quick_ticket/implement_now remain as bypass shortcuts.

4. Risks & constraints

• systemd: drain runs inside existing mc_pickup.py — no new service.
• CCGram telegram lock: persistent_luci.send() injects into the existing tmux session — no new claude subprocess, no new polling, no 409 risk.
• Single Flask app / mobile-friendly: new endpoints in app.py; composer response area plain text/markdown, no xterm.js.
• Context saturation: existing token watchdog (SOFT 700k / HARD 900k) handles rotation; digests must be compact; batch ≤20/drain.
• Self-heal loop: tag self-heal inbox items, dedupe max 1 per task_id per 30min.
• Distinct from rejected plan: no per-ticket tmux, no xterm.js, no worktrees, no new systemd unit — only wiring to existing luci-persistent + persistent_luci/send.

Summary

Recommended: Option B (P1 composer) + Option A (P2-3 scheduler). 3 required phases + 1 optional. ~5 days total, each phase independently shippable. Files: session_composer.html, app.py, scheduler.py, mc_pickup.py, models.py (schema), persistent-session context.