Data Flows Overview

This section documents how data moves through Luci's systems -- from external sources, through processing pipelines, into storage, and out to notifications.

Flow Index

Flow	Schedule	Source	Destination
12-data-flows/morning-briefing	06:00 UTC weekdays	Calendar APIs + MC tickets	Telegram DM
12-data-flows/email-pipeline	07:00 UTC weekdays	MS Graph API	email.db

Common Patterns

Fetch-Process-Store-Notify

Most data flows follow this pattern: 1. Fetch from an external API (Graph API, Google Calendar, etc.) 2. Process -- filter noise, classify, summarize 3. Store in a database (email.db, whatsapp-messages.db, mc.db) 4. Notify via Telegram when relevant

Scheduled via Scheduler

All recurring data flows are defined as scheduler tasks in /home/lucienne/workspace/tasks/. They run via scheduler.py tick (triggered by cron every minute). See 03-scheduler/overview for details.

Key Databases

Database	Path	Owner	Contents
email.db	`~/workspace/email.db`	Luci	Classified email metadata + summaries
whatsapp-messages.db	`~/workspace/data/whatsapp/whatsapp-messages.db`	WhatsApp monitor	Captured WhatsApp messages
mc.db	`~/workspace/mission-control/mc.db`	Mission Control	Tickets, task runs, sessions, heartbeats, ticket_context (vault bridge)
vault.db	`~/workspace/PKA/Vault/vault.db`	Lucienne	Knowledge graph (files, edges, tags), FTS5, activity log

Key Takeaways

Data flows are scheduler-driven -- each one is a task file in ~/workspace/tasks/
Most flows follow the fetch-process-store-notify pattern
External APIs (Graph, Google, WhatsApp) feed into local SQLite databases
Telegram is the universal notification channel for all flow outputs

Wiki