MC-4246 — [watchdog:missed:tailscale-watch] Task didn't fire: tailscale-watch

Home Board MC-4246

[watchdog:missed:tailscale-watch] Task didn't fire: tailscale-watch

Task `tailscale-watch` (Tailscale UDP/DERP health watch) has a cron schedule of `*/30 * * * *`. The previous expected fire was at **2026-05-26 13:30 UTC+02:00** (more than 15 mi...

State Done Next Action Closed Owner Luci Runtime Closed Age 18d ago

Description

MC-4246

Task `tailscale-watch` (Tailscale UDP/DERP health watch) has a cron schedule of `*/30 * * * *`. The previous expected fire was at **2026-05-26 13:30 UTC+02:00** (more than 15 min ago), but `task_runs` has no entry since then. Last actual run: **2026-05-26 12:30:26** Task file: `~/workspace/tasks/tailscale-watch.md` Investigate: is the task enabled? Is the schedule correct? Is there a lock or prerequisite blocking it?

Activity

done

Luci is working...

Details — Done · Medium · Luci ▼

State

Done

Closed

State

Priority

People

Owner (assigned to)

Controller

Timing / Details

Due Date

Snooze

Source scheduler-watchdog (scheduled)

Project infra

Created 18d ago

Updated 18d ago

Advanced / Operator evidence

Routing owner

Routes via

Operator console

Evidence

Ticket is done; runtime is closed. ticket_marked_done profile claude_opus_1m_high · cwd /home/lucienne/workspace · uptime 17d 20h · last activity 17d 18h ago MC is visibility-only. Hermes Luci launches and gates work outside MC, then mirrors evidence/status here. Raw console: luci · claude/anthropic/opus[1m]

Workflow

Start Dev Review + QA ▾

Select phases to include:

Research (scott) Implement (larry) (required) Council Review (council) Code Review (luci) Validate (tessa) (required) Sign-off (atlas)

Agents

Review Gates

Decision

WAT routing: choose an agent, review gate, or decision. Buttons use the live runtime when one is attached.

Luci 18d ago

REVIEW: tailscale-watchnot broken —ranfineat13:51:36.Missed13:00/13:30fireswerecollateralfroma41-minschedulerstall(12:58:03–13:39:07). Root cause: sa-airline-cpa-watch (id 307987) ran 2464s under the scheduler flock, blocking every other task. Its timeout: 3600 permits up to 1h of blocking. Schedule is 0 6 * * * but retry=true, so it likely got retried into the 12:58tick. State now: ⏵⏵bypasspermissionson (shift+tabtocycle)97375tokens - tailscale-watch: healthy, fired 13:51:36, state file fresh ⏵⏵bypasspermissionson (shift+tabtocycle)97375tokens -Scheduler:recovered,ticksnormal - task config: enabled=true, schedule */30 * * * * correct ⏵⏵bypasspermissionson (shift+tabtocycle)97375tokens -tailscale-watch:healthy,fired13:51:36,statefilefresh Scheduler: reoverd, ticks normal task config: enabl=true, schedue*/30 * * * *correct Recommendation (separate ticket, not this one):long-runningtasksshouldn'tholdtheglobalschedulerflock.Options:detachsa-airline-cpa-watchtoitsownsystemdunit,orcapitstimeoutto≤5minandoffloadheavyworktoa background job. This will recur on any task that nears its timeout. ⏵⏵bypasspermissionson (shift+tabtocycle)97375tokens This watchdog ticket can be closed — no tailscale fix needed. ⏵⏵bypasspermissionson (shift+tabtocycle)97375tokens {"mc_signal":"REVIEW","ticket":"MC-4246","nonce":"MC_TICKET_TURN:4246:1779796300105:3e755d99","payload":"tailscale-watch healthy; missed fires were collateral from sa-airline-cpa-watch holding scheduler flock 2464s (12:58-13:39).Recommendisolatinglongtasks."}

api 18d ago

Closed (orchestrator). Collateral from same incident as MC-4245: sa-airline-cpa-watch held flock 2464s (12:58–13:39), missed 13:00+13:30 fires. tailscale-watch itself healthy. Worker's recommendation aligns with MC-4216 (agent-watch W22) borrow proposal: 'Scheduler parallelism (maxConcurrentRuns=8)'. Surfaced to Elmar this morning for roadmap triage. Until that lands, alternative quick fix: detach sa-airline-cpa-watch to its own systemd unit OR cap timeout ≤5min + offload heavy work. Will recur on any task nearing its timeout under serial flock.