You're offline — showing cached data

Persistent Luci watchdog — liveness keep-alive + context escalation ladder

persistent-luci-watchdog
2026-06-13 06:11:41 SAST
← Back to Tasks

OVERRIDES runtime profile: uses direct_python (plain Python, no model) because the command chain never invokes the claude CLI or any LLM API — pure-infra task; scheduler provider env injection is a no-op (MC-4942 U12 sweep).

Runs every 5 minutes. Two independent jobs each tick:

1. Liveness keep-alive (MC-3763). Checks persistent_luci.status(). If the luci-persistent orchestrator session is dead, restarts it via ensure_started() and Telegram-alerts. A 2-strike grace (restart only on the 2nd consecutive offline observation) avoids fighting the daily 03:00 rotation while the fresh session boots. The strike counter is persisted to /tmp/luci-persistent-keepalive.json so systemd_watchdog.py corroborates orchestrator health hourly from the same source. Closes the post-OOM / post-reboot dead window from ~30 min (lazy heal) to ~5–10 min.

2. Context escalation ladder. Captures the luci-persistent tmux pane, parses the token counter from the Claude Code TUI footer, and escalates:

Cooldown is 5 minutes by default but shortens to 2 minutes above the soft threshold so a fast climb gets enough attempts to escalate. Logs every decision to ~/workspace/logs/persistent-luci-watchdog.log.

Implemented for MC-2364 after Persistent Luci hit 201.2k tokens on 2026-04-28 and stopped responding cleanly. Liveness keep-alive added for MC-3763 after the orchestrator-robustness eval found the session had no real liveness supervisor.

Every 5 minutes*/5 * * * *
Yes
shell
120s
No
in 3m (2026-06-13 06:15)

Task Definition

tasks/persistent-luci-watchdog.md
shell
set
No
No
python3 /home/lucienne/workspace/scripts/persistent_luci_watchdog.py

Edit Schedule

Result:
2015
Runs (7d)
2015
Completed
0
Failed
0
Timeout
100%
Success (7d)
0s
Avg Duration
infra luci persistent watchdog liveness

Run History (50)

StartedFinishedStatusDurationOutputLog
2m ago 2m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
7m ago 7m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
11m ago 11m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
17m ago 17m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
22m ago 22m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
27m ago 27m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
32m ago 32m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
37m ago 37m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
41m ago 41m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
47m ago 47m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
52m ago 52m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
57m ago 57m ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
1h ago 1h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
1h ago 1h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
1h ago 1h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
1h ago 1h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
1h ago 1h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
1h ago 1h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
2h ago 2h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
3h ago 3h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
4h ago 4h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
4h ago 4h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
4h ago 4h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
4h ago 4h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
4h ago 4h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
4h ago 4h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
4h ago 4h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log
4h ago 4h ago completed 0s
luci-persistent OK {"keepalive": {"ok": true, "action": "noop", "reason": "online", "online": true}, "watchdog": {"ok": ...
View log

Comments (0)

No comments yet — add instructions for Luci
Help