WhatsApp Monitoring

WhatsApp monitoring captures messages from Elmar's WhatsApp account using the Baileys library and stores them in a SQLite database for search and context retrieval.

Architecture

• Script: /home/lucienne/workspace/whatsapp/whatsapp-monitor.js (Node.js)
• Library: @whiskeysockets/baileys -- a reverse-engineered WhatsApp Web client (no Puppeteer/browser needed)
• Auth: Multi-file auth state stored at /home/lucienne/workspace/data/whatsapp/auth/ -- shares OpenClaw's session, so no QR scan needed if already authenticated
• Service: Runs as a systemd user service (whatsapp-monitor.service)
• PID lock: /tmp/whatsapp-monitor.pid prevents duplicate instances

How It Works

1. Connection: Baileys opens a WebSocket connection to WhatsApp servers using stored auth credentials
2. Message reception: Incoming and outgoing messages trigger the logMessage() handler
3. Filtering: Messages are checked against whatsapp-config.json rules:
4. Groups are excluded by default; only whitelisted groups pass through
5. Newsletters, LIDs, and status broadcasts are always excluded
6. Excluded contacts, patterns, and noise keywords are filtered out
7. Deduplication: In-memory set of 500 most recent message IDs prevents double-logging
8. Content extraction: Handles text, images (with caption), video, documents, audio, location, contacts, polls, and reactions
9. Persistence: Messages are written to both JSONL log and SQLite database
10. Reminder extraction: Messages with meaningful content are passed to a reminder extractor (regex + LLM-based)

Data Storage

SQLite Database

• Path: /home/lucienne/workspace/data/whatsapp/whatsapp-messages.db
• Mode: WAL (Write-Ahead Logging) for concurrent read safety
• Table: whatsapp_messages
• record_id (PK) -- message ID or SHA1 hash fallback
• message_id, timestamp, chat_id, chat_name
• sender_id, sender_name, body, direction
• from_me, is_group, conversation_id
• media_type, caption, raw_json
• FTS index: whatsapp_messages_fts (FTS5) -- indexes body, caption, sender_name, chat_name for full-text search

JSONL Log

• Complete log: /home/lucienne/workspace/logs/whatsapp/whatsapp-complete.log -- all messages that pass filters
• Filtered log: /home/lucienne/workspace/logs/whatsapp/whatsapp-filtered.log -- messages that were excluded (truncated)
• Acks log: /home/lucienne/workspace/logs/whatsapp/whatsapp-acks.log -- delivery acknowledgements
• Status file: /home/lucienne/workspace/logs/whatsapp/whatsapp-status.json -- current connection state + last message info

On startup, if the SQLite database is empty, it backfills from the JSONL complete log.

Configuration

• Path: /home/lucienne/workspace/data/whatsapp/whatsapp-config.json
• Included groups (11): Conradies - Blessed, Padel OG, Legal-Mind, 12JVCC, T5, Life Group, BARSA Members Group, Camino's wine and walk, Disrupt Operations, Commercial Notifications, Technical Group
• All personal contacts included (includeAllContacts: true)
• All groups excluded by default (excludeAllGroupsByDefault: true)
• Pending groups: BKCamino (auto-add by name pattern), plus any added by name pattern for auto-matching when the group ID is unknown
• Retention: 4 days (configured, enforced externally)

Health Check

• Script: /home/lucienne/workspace/scripts/whatsapp_health_check.py
• Schedule: Every 15 minutes (scheduler task whatsapp-health-check)
• Checks:
• systemd service status (is it running?)
• Database freshness (is the latest message recent enough?)
• Stale thresholds (adaptive):
• Daytime: 6 hours (360 min)
• Night (22:00-10:00 SAST): 14 hours (840 min)
• Weekend / SA public holidays: 20 hours (1200 min)
• Output: Writes /home/lucienne/workspace/data/whatsapp/health-status.json for dashboard consumption
• Alerting: Exits with code 1 on unhealthy state; the scheduler's notify_on: failure triggers a Telegram alert

Memory Management

• Max heap threshold: 200 MB -- triggers graceful restart if exceeded
• Heap usage logged periodically
• Node started with --max-old-space-size=512

Outbox (Sending Messages)

The monitor also supports outbound messaging via a JSONL outbox file: - Outbox: /home/lucienne/workspace/data/whatsapp/whatsapp-outbox.jsonl - Results: /home/lucienne/workspace/data/whatsapp/whatsapp-outbox-results.jsonl

Related Files

Key Takeaways

• Baileys provides a direct WebSocket connection to WhatsApp -- no browser automation needed
• Messages are stored in both JSONL (append-only, backward-compatible) and SQLite (queryable with FTS5)
• Group messages are opt-in only; personal contacts are all included
• Health checks run every 15 minutes with time-of-day-aware staleness thresholds
• The monitor shares auth with OpenClaw -- re-authentication is rarely needed