Worker System

Workers are background Claude Code sessions that pick up and execute MC tickets.

Architecture

Scheduler (every 1 min)
  └── mc_pickup.py
        ├── Fetch todo tickets assigned to Luci
        ├── Check available worker slots (MAX_WORKERS = 2)
        └── For each ticket:
              ├── Spawn: claude -p <prompt> --worktree mc-mc-<id> --effort max
              ├── Stream output via stream-json
              ├── Tee raw stream-json to mc-worker-<id>-stream.log (MC-747)
              ├── Parse DONE:/REVIEW:/QUESTION: prefix
              ├── Store tool_use/tool_result/text events in ticket_events table
              ├── Push ticket_event over SSE for live UI panel
              ├── Post result as comment on ticket
              └── Update ticket status

Key Files

~/workspace/mc_pickup.py — dispatcher + worker logic
~/.claude/settings-worker.json — worker-specific Claude settings (no Telegram plugin)
~/workspace/scripts/mc_hook.py — PostToolUse/Stop hooks for workers
~/workspace/scripts/queue_reaper.py — reaper for stuck queued_messages (MC-462)
~/workspace/tasks/queue-reaper.md — scheduler task: runs queue_reaper every 15 min

Working Directory Contract

Luci is anchored at ~/workspace, but workers are not all launched from there. The dispatcher and scheduler use ~/workspace as the machine-level root; each worker then resolves its own current working directory from the ticket project.

Ticket/project type	Worker cwd	Why
`General`, empty project, `PKA`	`~/workspace/PKA`	PKA repo-local instructions, skills, wiki, docs, and vault paths apply
`Mission Control`, `mission-control`, `MC`	`~/workspace/mission-control`	MC app code, templates, static assets, tests, and `mc.db` live here
`Infrastructure`, `Life`, `Finance`, `Machine Ops`	`~/workspace`	Machine-level scripts, tasks, logs, wiki, reports, and operational glue live here
Any project with a matching slug directory	`~/workspace/<project-slug>`	Lets project repos carry their own local context
Unknown project fallback	`~/workspace`	Safe machine-level fallback when no repo can be resolved

This distinction matters because ~/workspace/wiki is Luci's operational wiki, while ~/workspace/PKA and ~/workspace/mission-control are separate repos with their own local instructions. Workers can still read wiki pages by absolute path from any cwd, for example ~/workspace/wiki/02-mission-control/worker-system.md.

Mission Control itself is launched by systemd with WorkingDirectory=/home/lucienne/workspace/mission-control, but app.py sets WORKSPACE = Path.home() / "workspace". That is why /wiki renders ~/workspace/wiki, even though the Flask app process starts inside the MC repo.

Worker Lifecycle

Pickup: ticket-pickup scheduler task runs mc_pickup.py every minute
Dispatch: checks for todo tickets assigned to active MC worker roles (luci, tessa, scott, atlas, plus lucienne as Luci alias), respects MAX_WORKERS (2), and spawns or routes the relevant runtime
Isolation: each worker gets its own git worktree (--worktree mc-mc-<id>)
Effort: all workers run with --effort max for deepest reasoning
Prompt: includes KEY FACTS (server identity, services), ticket context, all comments
Role prompt: Luci coordinates, Tessa validates, Scott researches, Atlas reviews architecture/sign-off. Implementation work invokes dev-loop; QA/research/architecture phases use their role workflow instead.
Output: worker outputs DONE:/REVIEW:/QUESTION: prefix
Commit: Stop hook auto-commits + auto-pushes any changes
Cleanup: worktree cleaned up after session ends

Output Prefixes

Prefix	Ticket Status	Meaning
DONE:	done	Work completed, ticket closed
REVIEW:	in_review	Needs Elmar's review before closing
QUESTION:	needs_input	Blocked, needs Elmar's input
(no prefix)	in_review	Default — needs manual review

Downgrade Rule

If a worker posts REVIEW: or QUESTION: mid-session but then outputs DONE:, the DONE is downgraded to REVIEW. The first "needs input" signal wins.

Session Persistence

Fresh tickets get a random UUID session
Resume tickets (needs_input → Elmar answered) use --resume <stored_session_id>
After 15 turns, session rotates (fresh UUID with recent comment context)
Fork session flag when resuming from crash (not answering a question)
If Claude returns a different session_id in the result stream, MC treats that value as authoritative: it closes the old worker runtime row, opens a new row keyed to the actual session id, and records a new ticket attempt with the fork/cold-start reason. This prevents later replies from resuming the wrong branch.

Larry Dispatch

Tickets assigned to Larry are dispatched via SSH:

ssh elmar@46.225.208.1 "cd /home/elmar/cowork/projects/legalmind-explorer && claude -p ..."

Larry is intentionally limited to claude or codex as first-class CLIs. The post-completion review uses the selected Larry CLI/model instead of silently hardcoding Claude. Clean or skipped review now queues Tessa validation via the workflow action instead of marking the ticket done; critical review findings return the ticket to Larry for fixes.

Workflow Handoffs

Workflow child tickets feed the parent ticket forward automatically instead of completing as isolated records:

Larry implementation completes and queues send_to_tessa.
Tessa validation completes and queues send_to_luci_review.
Scott research completion also queues send_to_luci_review.
Luci review can output DONE: APPROVE, REVIEW: RETURN_FOR_FIXES, or REVIEW: SIGNOFF_REQUIRED to close, return to Larry, or queue Atlas.
Atlas sign-off can output DONE: APPROVE or REVIEW: RETURN_FOR_FIXES.

MC remains the workflow state machine. The tmux/CLI transcript is evidence and an attach surface, not the source of truth for parent/child workflow state.

Hooks

PostToolUse: mc_hook.py updates MC ticket activity label (Reading, Writing, etc.)
Stop: mc_hook.py auto-commits, auto-pushes, cleans up agent count file
PostToolUse (formatter): format_hook.py runs ruff format on .py files
PreCompact: precompact_hook.py saves worker state to memory before compression

Queue Reaper (MC-462)

scripts/queue_reaper.py runs every 15 min via the queue-reaper scheduler task. It handles stuck queued_messages that workers failed to consume:

Expire: messages older than 30 min without being claimed are expired
Retry: failed messages are retried up to 3 times
Dead-letter: messages that fail 3+ times are marked as permanently failed
Alert: sends Telegram notification when messages are stuck or expired

This prevents the "silent message death" problem where user replies like "yes, go ahead" would sit unprocessed indefinitely.

Concurrency

MAX_WORKERS = 2 (hard cap) — lowered 3→2 (MC-3765) for OOM headroom on the memory-starved box; each heavy claude[1m] worker is ~300-400MB RSS
OOM bias (MC-3765): mc_tmux.start_session calls apply_oom_score() — luci-persistent orchestrator gets oom_score_adj=-800 (killed last), mc-* worker sessions get +600 (killed first). MC gunicorn carries OOMScoreAdjust=-600 via systemd drop-in luci-dashboard.service.d/oom-protect.conf
active_workers_by_db() counts in_progress tickets with live PIDs
Orphan cleanup: if PID is dead, ticket reset to todo
Worker slots checked before each dispatch cycle

Live Activity Panel (MC-747)

Workers emit structured events to ticket_events as they run, visible in the ticket detail UI:

tool_use events: fired at content_block_stop with tool name, context string, and key args
tool_result events: fired on each tool_result stream event (content truncated to 500 chars)
text events: fired at turn completion with final assistant output (tail 500 chars)
result events: fired at turn end with duration_s and cost_usd

Raw stream-json lines are also tee'd to ~/workspace/logs/mc-worker-<id>-stream.log for archival and debugging. The print-log (mc-worker-<id>.log) continues unchanged.

On the UI side: the ticket detail page calls GET /api/v1/tickets/<identifier>/events on load to replay history, then subscribes to the per-ticket SSE stream (/api/v1/tickets/<id>/stream) for live ticket_event messages. The panel is collapsed by default and auto-opens when the ticket is in_progress.

Session Auto-Resume (MC-3249)

When a worker session is interrupted (e.g. context window exhaustion, OOM, rate-limit cascade), the scheduler detects the crash and auto-resumes:

Interrupted workers with stored session_id get --resume on next pickup cycle
Fresh context is injected with recent comments so the worker continues seamlessly
This replaces the older "reset to todo and start over" behavior — workers now pick up where they left off
Fork detection: if Claude returns a different session_id, MC records a fork/cold-start reason

Auth Env-Leak Fix (MC-3412)

Worker sessions spawned by mc_pickup.py now clear stale ANTHROPIC_* environment variables before launch. Previously, leftover API keys from prior sessions could leak into the worker subprocess, causing authentication confusion when the worker's intended provider was non-Anthropic (e.g. GLM, Kimi). The fix strips these vars in mc_pickup.py before Popen.

Stale Runtime Reaping (MC-3536)

Orphaned interactive ticket runtimes (tmux sessions for in_progress tickets whose PID no longer exists) are now detected and cleaned up automatically. The reaper:

Scans active runtimes in mc.db where status is in_progress
Checks if the associated tmux PID is still alive via /proc/<pid>/
Dead PIDs trigger runtime row closure + ticket status reset
Prevents ghost "in progress" tickets that block worker slots

MC Pickup Resilience (MC-3552)

The mc_request() helper in mc_pickup.py now handles transient gunicorn worker-recycle blips (connection refused, 502) with a short retry loop instead of immediately failing the pickup cycle. This prevents false-negative "MC unreachable" errors during gunicorn worker rotation, which was causing pickup to skip valid tickets.

Key Takeaways

Workers are role-aware: Luci/Tessa/Scott/Atlas tickets are picked up by the scheduler instead of becoming passive workflow records
Workflow handoffs are role-aware: Larry queues Tessa, Tessa/Scott queue Luci review, and Luci/Atlas output controls approve/return/sign-off
dev-loop skill is mandatory for implementation work; QA/research/architecture phases use their role-specific workflow
Worktrees prevent edit conflicts between workers
Worker cwd is resolved by project: PKA/general work runs in ~/workspace/PKA, MC work in ~/workspace/mission-control, and machine ops in ~/workspace
Worker session identity is MC-tracked, but the CLI result stream wins if it reports a forked or changed session id
Stop hook ensures commits and pushes happen even on crash
MAX_WORKERS=2 + main/persistent session = 3 concurrent active sessions before Larry remote work
529 overload errors tracked via api_error_tracker.py
queue_reaper.py prevents stuck messages (expires >30min, retries 3x, alerts via Telegram)
vault_mcp.py auto-logs consulted vault files to ticket_context during worker sessions
Live activity panel (MC-747) shows tool calls in real-time — no more SSH+tail to watch a worker

Wiki