Observability sweep — stuck tickets + orphan tmux

← Back to Tasks

OVERRIDES runtime profile: uses notify.py (urllib direct Telegram) + direct sqlite read + tmux ls. No claude CLI invocation.

Every 15 min, two checks (MC-3370):

Stuck-ticket alerter — diffs /api/v1/tickets/stuck against last-seen set on disk, Telegram-alerts only NEW entries (1h dedup per ticket). Closes the gap where get_stuck_tickets() was pull-only and waited for a human to open the page.
Orphan tmux sweep — diffs tmux ls (mc-MC-* sessions) against runtime_sessions WHERE status IN ('starting','running','idle','needs_input'). Flags tmux sessions with no active row (leaked browser/CLI processes) and active rows with no tmux session (stale db). 6h dedup per orphan key.

Implementation: ~/workspace/scripts/observability_sweep.py. Always exits 0. Parent ticket: MC-3365.

Schedule Every 15 minutes*/15 * * * *

EnabledYes

Run Asshell

Timeout120s

RetryNo

Next Runin 5m (2026-06-13 06:15)

Task Definition

tasks/observability-sweep.md

Run Asshell

Commandset

Tool-lessNo

BackgroundNo

Runtime profile

Notify destination

. ~/.claude/env/api_keys.env 2>/dev/null; python3 /home/lucienne/workspace/scripts/observability_sweep.py

---
id: observability-sweep
title: Observability sweep — stuck tickets + orphan tmux
schedule: "*/15 * * * *"
timeout: 120
retry: false
enabled: true
notify_on: failure
run_as: shell
command: ". ~/.claude/env/api_keys.env 2>/dev/null; python3 /home/lucienne/workspace/scripts/observability_sweep.py"
tags: [infra, observability, watchdog]
runtime_profile: direct_mixed
---

**OVERRIDES runtime profile:** uses notify.py (urllib direct Telegram) +
direct sqlite read + `tmux ls`. No `claude` CLI invocation.

Every 15 min, two checks (MC-3370):

1. **Stuck-ticket alerter** — diffs `/api/v1/tickets/stuck` against last-seen
   set on disk, Telegram-alerts only NEW entries (1h dedup per ticket).
   Closes the gap where `get_stuck_tickets()` was pull-only and waited for a
   human to open the page.
2. **Orphan tmux sweep** — diffs `tmux ls` (mc-MC-* sessions) against
   `runtime_sessions WHERE status IN ('starting','running','idle','needs_input')`.
   Flags tmux sessions with no active row (leaked browser/CLI processes) and
   active rows with no tmux session (stale db). 6h dedup per orphan key.

Implementation: `~/workspace/scripts/observability_sweep.py`. Always exits 0.
Parent ticket: MC-3365.

Edit Schedule

Frequency

Result:

694

Runs (7d)

694

Completed

Failed

Timeout

100%

Success (7d)

Avg Duration

infra observability watchdog

Run History (50)

Started	Finished	Status	Duration	Output	Log
9m ago	9m ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
25m ago	25m ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
40m ago	40m ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
55m ago	55m ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
1h ago	1h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
1h ago	1h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
2h ago	2h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
2h ago	2h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
2h ago	2h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
2h ago	2h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
3h ago	3h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
3h ago	3h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
3h ago	3h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
3h ago	3h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
4h ago	4h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
4h ago	4h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
4h ago	4h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
4h ago	4h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
5h ago	5h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
5h ago	5h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
5h ago	5h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
5h ago	5h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
6h ago	6h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
6h ago	6h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
6h ago	6h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
6h ago	6h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
7h ago	7h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
7h ago	7h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
7h ago	7h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
7h ago	7h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
8h ago	8h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
8h ago	8h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
8h ago	8h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
8h ago	8h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
9h ago	9h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
9h ago	9h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
9h ago	9h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
9h ago	9h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
10h ago	10h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
10h ago	10h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
10h ago	10h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
10h ago	10h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
11h ago	11h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
11h ago	11h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
11h ago	11h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
11h ago	11h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
12h ago	12h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
12h ago	12h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
12h ago	12h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log
12h ago	12h ago	completed	0s	[obs-sweep] stuck tickets reported: 0 [obs-sweep] tmux=0 db_active=0 tmux_orphans=0 stale_rows=0	View log

Comments (0)

No comments yet — add instructions for Luci

←