Tailscale UDP/DERP health watch

← Back to Tasks

OVERRIDES runtime profile: uses direct_python because this network health watch runs deterministic Tailscale checks and does not invoke Claude/Codex or any LLM API.

Half-hourly health check on Tailscale's NAT-traversal layer.

Background: 2026-05-07 SAST — Tailscale on Luci has broken UDP NAT discovery (netcheck reports UDP: false, IPv4: no addr found, no DERP latency). Phone-to-server traffic relays via DERP-jnb with 241ms-2.1s RTT, making the mobile PWA feel unusable. Workaround in place: phone uses public IP http://204.168.188.33:3001 direct. SSH and admin paths still work fine via DERP since lower-frequency traffic tolerates the latency.

Full diagnosis: ~/workspace/reports/tailscale-degraded-2026-05-07.md

Behaviour of this watcher: - Run tailscale netcheck and tailscale ping elmars-s26-ultra - Classify state: healthy | degraded | broken - Persist a rolling history to ~/workspace/state/tailscale_health.json - Alert (force=True, bypasses quiet hours): - Recovery: when state flips from degraded → healthy - Warn: degraded continuously for ≥24h - Escalate: degraded continuously for ≥7d, recommend Cloudflare Tunnel - Each alert rate-limited to once per 24h per key

Parent MC ticket: MC-2963.

Schedule Every 30 minutes*/30 * * * *

EnabledYes

Run Asshell

Timeout60s

RetryNo

Next Runin 21m (2026-06-13 06:30)

Task Definition

tasks/tailscale-watch.md

Run Asshell

Commandset

Tool-lessNo

BackgroundNo

Runtime profile

Notify destination

python3 /home/lucienne/workspace/scripts/tailscale_watch.py

---
id: tailscale-watch
title: Tailscale UDP/DERP health watch
schedule: "*/30 * * * *"
timeout: 60
retry: false
enabled: true
notify_on: failure
notify_to: dm
run_as: shell
command: "python3 /home/lucienne/workspace/scripts/tailscale_watch.py"
tags: [infrastructure, tailscale, monitoring]
runtime_profile: direct_python
---

**OVERRIDES runtime profile:** uses `direct_python` because this network health
watch runs deterministic Tailscale checks and does not invoke Claude/Codex or
any LLM API.

Half-hourly health check on Tailscale's NAT-traversal layer.

Background: 2026-05-07 SAST — Tailscale on Luci has broken UDP NAT discovery
(`netcheck` reports `UDP: false`, `IPv4: no addr found`, no DERP latency).
Phone-to-server traffic relays via DERP-jnb with 241ms-2.1s RTT, making the
mobile PWA feel unusable. Workaround in place: phone uses public IP
`http://204.168.188.33:3001` direct. SSH and admin paths still work fine via
DERP since lower-frequency traffic tolerates the latency.

Full diagnosis:
~/workspace/reports/tailscale-degraded-2026-05-07.md

Behaviour of this watcher:
- Run `tailscale netcheck` and `tailscale ping elmars-s26-ultra`
- Classify state: healthy | degraded | broken
- Persist a rolling history to `~/workspace/state/tailscale_health.json`
- Alert (force=True, bypasses quiet hours):
  - Recovery: when state flips from degraded → healthy
  - Warn: degraded continuously for ≥24h
  - Escalate: degraded continuously for ≥7d, recommend Cloudflare Tunnel
- Each alert rate-limited to once per 24h per key

Parent MC ticket: MC-2963.

Edit Schedule

Frequency

Result:

349

Runs (7d)

349

Completed

Failed

Timeout

100%

Success (7d)

Avg Duration

infrastructure tailscale monitoring

Run History (50)

Started	Finished	Status	Duration	Output	Log
8m ago	8m ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=878.67	View log
39m ago	39m ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=878.16	View log
1h ago	1h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=877.73	View log
2h ago	2h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=877.16	View log
2h ago	2h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=876.73	View log
3h ago	3h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=876.18	View log
3h ago	3h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=875.67	View log
4h ago	4h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=875.16	View log
4h ago	4h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=874.7	View log
5h ago	5h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=874.17	View log
5h ago	5h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=873.67	View log
6h ago	6h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=873.16	View log
6h ago	6h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=872.67	View log
7h ago	7h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=872.16	View log
7h ago	7h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=871.69	View log
8h ago	8h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=871.16	View log
8h ago	8h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=870.67	View log
9h ago	9h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=870.16	View log
9h ago	9h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=869.66	View log
10h ago	10h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=869.16	View log
10h ago	10h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=868.66	View log
11h ago	11h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=868.16	View log
11h ago	11h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=867.72	View log
12h ago	12h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=867.17	View log
12h ago	12h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=866.68	View log
13h ago	13h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=866.17	View log
13h ago	13h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=865.71	View log
14h ago	14h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=865.17	View log
14h ago	14h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=864.69	View log
15h ago	15h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=864.16	View log
15h ago	15h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=863.66	View log
16h ago	16h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=863.17	View log
16h ago	16h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=862.67	View log
17h ago	17h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=862.17	View log
17h ago	17h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=861.67	View log
18h ago	18h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=861.16	View log
18h ago	18h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=860.81	View log
19h ago	19h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=860.16	View log
19h ago	19h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=859.66	View log
20h ago	20h ago	completed	10s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=None hours_degraded=859.16	View log
20h ago	20h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=858.66	View log
21h ago	21h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=858.16	View log
21h ago	21h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=857.66	View log
22h ago	22h ago	completed	8s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=857.16	View log
22h ago	22h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=856.71	View log
23h ago	23h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=856.16	View log
23h ago	23h ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=855.66	View log
24h ago	24h ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=855.16	View log
1d ago	1d ago	completed	6s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=854.68	View log
1d ago	1d ago	completed	7s	tailscale-watch: state=degraded udp=False ipv4=None ping_via=derp hours_degraded=854.16	View log

Comments (0)

No comments yet — add instructions for Luci

←