You're offline — showing cached data

MC-4585

Vault vector embeddings: portable cache and cheaper API backend
2026-06-13 08:47:41 SAST
Home Board MC-4585

Vault vector embeddings: portable cache and cheaper API backend

Context from Iris/Elmar discussion: vault.db is local per machine, so rebuilding vector embeddings on every machine is wasteful and fragile. Current state on Luci/PKA: Vault/va...
State Done Next Action Closed Owner Luci Runtime Closed Age 11d ago
MC-4585
Ticket is done; runtime is closed. · profile claude_opus_1m_high · cwd /home/lucienne/workspace/PKA · uptime 11d 3h · last activity 10d 20h ago

Description

MC-4585
Context from Iris/Elmar discussion: vault.db is local per machine, so rebuilding vector embeddings on every machine is wasteful and fragile. Current state on Luci/PKA: Vault/vault.db vector layer is healthy after rebuild: vec_meta=848, vec_chunks=1120; scopes covered are Vault/memory/, SecondBrain/, and ~/.claude/vault/. data/ remains intentionally excluded because prior runs hit high RSS/OOM risk. Decision guidance: - If switching to API embeddings, prefer OpenAI text-embedding-3-small over Gemini: observed corpus ~843k tokens; OpenAI batch about $0.008/full re-embed, standard about $0.017, Gemini about $0.126. - However, the better architecture is a portable embedding cache keyed by chunk/content hash + model + dimension, not path, so each local vault.db can import vectors and only embed cache misses. - Keep exact ID-number searches on FTS/regex, not semantic vector search. Requested work for Luci: 1. Add/implement portable embedding cache or export/import layer for scripts/embed_memories.py. 2. Store/cache vectors by chunk_hash/content_hash, embedding_model, embedding_dim/provider; avoid absolute-path coupling. 3. If API backend is implemented, use OpenAI text-embedding-3-small with 768 dimensions unless explicitly directed otherwise; do not choose Gemini by default. 4. Ensure scheduled vault rebuild runs embed_memories.py after index.py --rebuild. 5. Keep data/ excluded unless Elmar explicitly approves broader scope. 6. Add verification/health check: embed_memories.py --dry-run returns 0 pending for intended scopes; vector coverage matches expected scopes. Acceptance evidence: - Files changed listed. - Command output showing index/re-embed/sync-cache verification. - Counts for files/search_fts/vec_meta/vec_chunks after verification. - Confirmation no raw data/ vector expansion was introduced.

Activity

done
Luci is working...
Live
No activity yet
Help