Email Pipeline

The email pipeline fetches Outlook emails via the MS Graph API, classifies them by project, generates summaries, and stores the results in email.db. See 09-integrations/email for the full integration reference.

Pipeline Overview

MS Graph API  -->  Fetch last 2 days of emails
      |
      v
inbox-rules.md  -->  Filter newsletters / noise
      |
      v
Gemini Flash  -->  Classify to project + summarize
      |
      v
email.db  -->  Insert metadata + summary
      |
      v
email_attachment_sync.py  -->  Download attachments (best-effort)

Trigger

• Task ID: email-sync
• Path: /home/lucienne/workspace/tasks/email-sync.md
• Schedule: 0 7 * * 1-5 (07:00 UTC = 09:00 SAST, weekdays)
• Timeout: 600 seconds
• Retry: enabled
• Execution: Spawns Claude Code (claude -p) with a structured prompt that reads the email-index skill and performs each step

Step-by-Step

1. Fetch Emails

• Uses graph_api.py at ~/workspace/scripts/graph_api.py
• Calls list-messages --since <2 days ago>
• Fetches up to 500 emails per run via Graph API pagination
• Selects: internetMessageId, conversationId, subject, from, receivedDateTime, body, hasAttachments

2. Filter Noise

• Reads rules from ~/workspace/cowork/context/inbox-rules.md
• Sender blocklist: Exact matches and wildcard patterns (e.g., *@zapiermail.com)
• Subject skip patterns: Specific subject keywords to auto-skip
• Filtered emails are silently dropped (not stored)

3. Classify and Summarize

• Each passing email is classified to a project from ~25 known project names
• A 1-sentence summary is generated
• Classification can be done by:
• Claude Code (in the daily sync task -- LLM-based classification)
• Gemini Flash (in the backfill script -- ~$0.01/day)
• Sender-to-project mapping (cached in .email-sender-map.json, built from existing email.db data)

4. Insert into email.db

• Uses the email-index skill's query.py insert command
• Database: ~/workspace/email.db (SQLite)
• Primary key: internet_message_id (RFC standard -- survives email moves/archive)
• No bodies stored -- only metadata + summary
• Stores: subject, sender, received date, project, summary, confidence, has_attachments

5. Attachment Sync (Best-Effort)

• After each successful insert, runs email_attachment_sync.py --message-id <graph_message_id>
• Downloads attachments via Graph API
• Failures are logged but do not fail the overall sync

Backfill Pipeline

A separate script handles enriching existing records:

• Script: ~/workspace/scripts/email_backfill.py
• Purpose: Fetch bodies and generate summaries for emails already in email.db that lack summaries
• Process:
• Query email.db for records with empty summaries
• Fetch body from Graph API via internet_message_id
• Send to Gemini Flash for classification + summary
• Update email.db
• Rate limiting: 1 second between Graph API calls, batched Gemini calls

Search (Downstream)

The email-index skill provides the query interface:

python ~/.claude/skills/email-index/query.py project "Project Heron"
python ~/.claude/skills/email-index/query.py search "MOI harith"
python ~/.claude/skills/email-index/query.py sender "stephan.spamer"
python ~/.claude/skills/email-index/query.py recent --limit 10
python ~/.claude/skills/email-index/query.py stats

All queries run on Luci via SSH over Tailscale. Access from other machines uses the email-index skill which handles the SSH connection.

State Management

Error Handling

• If Graph API token is expired, the sync auto-refreshes it
• If refresh fails, the task fails and notify_on: failure sends a Telegram alert
• Attachment sync failures are logged but do not block email processing
• The backfill script handles HTTP 401 errors gracefully (re-auth needed)

Key Takeaways

• The email pipeline runs daily at 09:00 SAST and processes the last 2 days of Outlook emails
• Noise filtering uses a rules file shared with the CoWork project
• Classification uses a combination of LLM judgment and cached sender-to-project mappings
• email.db stores only metadata and summaries -- full bodies are fetched on demand from Graph API
• The attachment sync is a separate best-effort step that runs after each email insert
• The internet_message_id is the stable primary key -- it never changes even when emails are moved or archived