MC-4359 — [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout)

Home Board MC-4359

[watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout)

Task `pool-health` finished with status **timeout** at 2026-05-28 06:30:25. Error (if captured): ``` reaper: orphaned at 2026-05-28T06:41:00.342931+02:00 — no live scheduler en...

State Done Next Action Closed Owner Luci Runtime Closed Age 16d ago

← MC-4359

Ticket is done; runtime is closed. · profile claude_opus_1m_medium · cwd /home/lucienne/workspace · uptime 16d 3h · last activity 16d 1h ago

Description

MC-4359

Task `pool-health` finished with status **timeout** at 2026-05-28 06:30:25. Error (if captured): ``` reaper: orphaned at 2026-05-28T06:41:00.342931+02:00 — no live scheduler entry (age=634s, threshold=600s) ``` This may be a one-off or a recurring issue — check previous runs in mc.db `task_runs` table.

Activity

done

Luci is working...

Details — Done · High · Luci ▼

State

Done

Closed

State

Priority

People

Owner (assigned to)

Controller

Timing / Details

Due Date

Snooze

Source scheduler-watchdog (scheduled)

Project infra

Created 16d ago

Updated 16d ago

Advanced / Operator evidence

Routing owner

Routes via

Operator console

Evidence

Ticket is done; runtime is closed. ticket_marked_done profile claude_opus_1m_medium · cwd /home/lucienne/workspace · uptime 16d 3h · last activity 16d 1h ago MC is visibility-only. Hermes Luci launches and gates work outside MC, then mirrors evidence/status here. Raw console: luci · claude/anthropic/opus[1m]

Workflow

Start Dev Review + QA ▾

Select phases to include:

Research (scott) Implement (larry) (required) Council Review (council) Code Review (luci) Validate (tessa) (required) Sign-off (atlas)

Agents

Review Gates

Decision

WAT routing: choose an agent, review gate, or decision. Buttons use the live runtime when one is attached.

Luci 16d ago

Root-caused MC-4359: pool-health run 324861 hung 634s in FetchCoordinator.force() — subprocess.run(timeout=60) kills the git child but blocks on captured pipes when git spawns an ssh/credential helper on a network stall. Hardened force() (GIT_TERMINAL_PROMPT=0 + stdin=DEVNULL + own process group + killpg on timeout). Committed 5dee58ff, pushed to slot1/mc-MC-4359 (awaiting merge to master). One-off — subsequent runs all ~1.7s.

luci 16d ago

Workflow action `review_ready` applied. Work ready for review.

orchestrator-board-sweep 16d ago

Auto-closed by orchestrator board-sweep: the underlying scheduled task `pool-health` has recovered — task_runs shows a successful run after this failure ticket was raised. The watchdog condition no longer holds.

luci 16d ago

Shadow reviewer verdict: UNCERTAIN. Needs human decision. Reasons: The worker claims to have root-caused and fixed the issue in commit 5dee58ff, but the diff is unavailable for inspection. There is no way to verify that the committed change actually implements the described hardening (GIT_TERMINAL_PROMPT=0, stdin=DEVNULL, own process group, killpg on timeout) or that it relates to the pool-health task at all. The orchestrator's auto-close based on task recovery does not confirm the code change was the cause. Gaps: - diff unavailable — cannot verify any code change was made or what it contains - cannot confirm the fix addresses the specific timeout/root-cause described - no test evidence or query results shown for the mc.db task_runs check

luci 16d ago

Luci picking up this ticket (background worker).

luci 16d ago

Ticket picked up by Luci via MC dispatcher. MC-4359: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) Work this ticket in the live tmux runtime. Use DONE:, REVIEW:, or QUESTION: when you need MC to reflect the next state.

luci 16d ago

[failed_to_inject] runtime_busy: Ticket runtime is not ready for input (status=running). Wait for the current turn to finish, or explicitly interrupt/restart the runtime before sending more terminal input. This message Ticket picked up by Luci via MC dispatcher. MC-4359: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) Work this ticket in the live tmux runtime. Use DONE:, REVIEW:, or QUESTION: when you need MC to reflect the next state.

luci 16d ago

Ticket runtime send failed while dispatching. root_cause: HTTP Error 409: CONFLICT safe_retry: returned to `todo` for a fresh runtime retry. human_input_required: no

system 16d ago

RUNTIME TERMINAL STATE (MC-3482 contract) status: warning summary: Ticket runtime send failed; parked for automatic recovery. root_cause: send failed while injecting into ticket runtime: Ticket runtime is not ready for input (status=running). Wait for the current turn to finish, or explicitly interrupt/restart the runtime before sending more terminal input. This message was not sent. safe_retry: Queued message was returned to pending and the ticket was returned to todo for a fresh runtime retry; no human input is needed. stop_condition: After the configured retry limit, leave the ticket blocked for operator inspection instead of looping. human_input_required: no next_actions: - Pickup will retry the pending message in a fresh runtime. - Inspect runtime/send only if the retry fails again. artifacts: - ticket:MC-4359

luci 16d ago

Luci picking up this ticket (background worker).

luci 16d ago

[failed_to_inject] runtime_busy: Ticket runtime is not ready for input (status=running). Wait for the current turn to finish, or explicitly interrupt/restart the runtime before sending more terminal input. This message [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout)

system 16d ago

RUNTIME TERMINAL STATE (MC-3482 contract) status: error summary: Ticket runtime send failed; parked for automatic recovery. root_cause: send failed while injecting into ticket runtime: Ticket runtime is not ready for input (status=running). Wait for the current turn to finish, or explicitly interrupt/restart the runtime before sending more terminal input. This message was not sent. safe_retry: Retry limit reached; operator/human inspection is required before another automatic runtime dispatch. stop_condition: After the configured retry limit, leave the ticket blocked for operator inspection instead of looping. human_input_required: yes next_actions: - Pickup will retry the pending message in a fresh runtime. - Inspect runtime/send only if the retry fails again. artifacts: - ticket:MC-4359

luci 16d ago

[failed_to_inject] runtime_busy: Ticket runtime is not ready for input (status=running). Wait for the current turn to finish, or explicitly interrupt/restart the runtime before sending more terminal input. This message [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout)

luci 16d ago

[system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout) [system]: Ticket moved to Todo: [watchdog:failed:pool-health:2026-05-28 06] Task failed: pool-health (timeout)

api 16d ago

Orchestrator gate PASS — landed to master (cherry-pick e75e6083 of slot1 5dee58ff). The shadow UNCERTAIN was the known diff-fetch tooling gap (MC-4341), not a defect — I inspected the diff directly. Fix is sound and scoped to FetchCoordinator.force(): start_new_session makes the git child a pgroup leader, killpg(SIGKILL) on TimeoutExpired reaps lingering ssh/credential helpers so captured pipes close, GIT_TERMINAL_PROMPT=0 + stdin=DEVNULL prevents the auth-prompt stall that caused the 634s hang. tests/test_worktree_pool.py 48 pass, py_compile clean. Composes with watchdog recovery-suppression (e063617c): this prevents the timeout; that suppresses the noise ticket if any residual failure recovers.

Live ▼

No activity yet

←