Data freshness watchdog — check input feeds, alert on stale

← Back to Tasks

OVERRIDES runtime profile: pure stdlib Python, no LLM calls — direct_no_llm. Hourly check of canonical input feeds (CEO dashboard, Findash, Exco, ACSA, IATA fuel, FlightAware, booking-curve, wiki pages, ccgram/MC services, MC DB).

Writes ~/workspace/state/data_freshness.json. MC /api/v1/health/data-freshness exposes it; dashboard / renders 'Data feeds' grid. Telegram fires on first stale event per feed per 24h with owner unit (LaunchAgent / systemd / scheduler task) named in body. 3+ stale at once → escalate priority critical, tag Lucienne. Silent on green.

Trigger: MC-2936. Recurring class of bug — pipeline silently stops, nobody notices for days (Exco MD pipeline 10-day silent failure 2026-05-07).

Schedule Every hour at :000 * * * *

EnabledYes

Run Asshell

Timeout120s

RetryNo

Next Runin 51m (2026-06-13 07:00)

Task Definition

tasks/data-freshness-check.md

Run Asshell

Commandset

Tool-lessNo

BackgroundNo

Runtime profile

Notify destination

cd /home/lucienne/workspace && python3 scripts/data_freshness_check.py

---
id: data-freshness-check
title: Data freshness watchdog — check input feeds, alert on stale
schedule: "0 * * * *"
timeout: 120
retry: false
enabled: true
notify_on: failure
runtime_profile: direct_no_llm
run_as: shell
command: "cd /home/lucienne/workspace && python3 scripts/data_freshness_check.py"
tags: [monitoring, watchdog, freshness, mc-2936]
---

**OVERRIDES runtime profile:** pure stdlib Python, no LLM calls — `direct_no_llm`. Hourly check of canonical input feeds (CEO dashboard, Findash, Exco, ACSA, IATA fuel, FlightAware, booking-curve, wiki pages, ccgram/MC services, MC DB).

Writes `~/workspace/state/data_freshness.json`. MC `/api/v1/health/data-freshness` exposes it; dashboard `/` renders 'Data feeds' grid. Telegram fires on first stale event per feed per 24h with owner unit (LaunchAgent / systemd / scheduler task) named in body. 3+ stale at once → escalate priority `critical`, tag Lucienne. Silent on green.

Trigger: MC-2936. Recurring class of bug — pipeline silently stops, nobody notices for days (Exco MD pipeline 10-day silent failure 2026-05-07).

Edit Schedule

Frequency

Result:

175

Runs (7d)

175

Completed

Failed

Timeout

100%

Success (7d)

Avg Duration

monitoring watchdog freshness mc-2936

Run History (50)

Started	Finished	Status	Duration	Output	Log
9m ago	9m ago	completed	4s	data-freshness: 9/21 stale	View log
1h ago	1h ago	completed	2s	data-freshness: 9/21 stale	View log
2h ago	2h ago	completed	2s	data-freshness: 9/21 stale	View log
3h ago	3h ago	completed	4s	data-freshness: 9/21 stale	View log
4h ago	4h ago	completed	4s	data-freshness: 9/21 stale	View log
5h ago	5h ago	completed	5s	data-freshness: 9/21 stale	View log
6h ago	6h ago	completed	2s	data-freshness: 7/21 stale	View log
7h ago	7h ago	completed	5s	data-freshness: 7/21 stale	View log
8h ago	8h ago	completed	2s	data-freshness: 7/21 stale	View log
9h ago	9h ago	completed	2s	data-freshness: 7/21 stale	View log
10h ago	10h ago	completed	0s	data-freshness: 7/20 stale	View log
11h ago	11h ago	completed	0s	data-freshness: 7/20 stale	View log
12h ago	12h ago	completed	0s	data-freshness: 7/20 stale	View log
13h ago	13h ago	completed	0s	data-freshness: 7/20 stale	View log
14h ago	14h ago	completed	0s	data-freshness: 7/20 stale	View log
15h ago	15h ago	completed	0s	data-freshness: 7/20 stale	View log
16h ago	16h ago	completed	0s	data-freshness: 7/20 stale	View log
17h ago	17h ago	completed	0s	data-freshness: 7/20 stale	View log
18h ago	18h ago	completed	0s	data-freshness: 7/20 stale	View log
19h ago	19h ago	completed	0s	data-freshness: 7/20 stale	View log
20h ago	20h ago	completed	0s	data-freshness: 7/20 stale	View log
21h ago	21h ago	completed	0s	data-freshness: 7/20 stale	View log
22h ago	22h ago	completed	0s	data-freshness: 7/20 stale	View log
23h ago	23h ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
1d ago	1d ago	completed	0s	data-freshness: 7/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log
2d ago	2d ago	completed	0s	data-freshness: 9/20 stale	View log

Comments (0)

No comments yet — add instructions for Luci

←