Vault vector embeddings: portable cache and cheaper API backend
Context from Iris/Elmar discussion: vault.db is local per machine, so rebuilding vector embeddings on every machine is wasteful and fragile. Current state on Luci/PKA: Vault/va...
StateDoneNext ActionClosedOwnerLuciRuntimeClosedAge11d ago
Ticket is done; runtime is closed.·profile claude_opus_1m_high · cwd /home/lucienne/workspace/PKA · uptime 11d 3h · last activity 10d 20h ago
Description
MC-4585
Context from Iris/Elmar discussion:
vault.db is local per machine, so rebuilding vector embeddings on every machine is wasteful and fragile. Current state on Luci/PKA: Vault/vault.db vector layer is healthy after rebuild: vec_meta=848, vec_chunks=1120; scopes covered are Vault/memory/, SecondBrain/, and ~/.claude/vault/. data/ remains intentionally excluded because prior runs hit high RSS/OOM risk.
Decision guidance:
- If switching to API embeddings, prefer OpenAI text-embedding-3-small over Gemini: observed corpus ~843k tokens; OpenAI batch about $0.008/full re-embed, standard about $0.017, Gemini about $0.126.
- However, the better architecture is a portable embedding cache keyed by chunk/content hash + model + dimension, not path, so each local vault.db can import vectors and only embed cache misses.
- Keep exact ID-number searches on FTS/regex, not semantic vector search.
Requested work for Luci:
1. Add/implement portable embedding cache or export/import layer for scripts/embed_memories.py.
2. Store/cache vectors by chunk_hash/content_hash, embedding_model, embedding_dim/provider; avoid absolute-path coupling.
3. If API backend is implemented, use OpenAI text-embedding-3-small with 768 dimensions unless explicitly directed otherwise; do not choose Gemini by default.
4. Ensure scheduled vault rebuild runs embed_memories.py after index.py --rebuild.
5. Keep data/ excluded unless Elmar explicitly approves broader scope.
6. Add verification/health check: embed_memories.py --dry-run returns 0 pending for intended scopes; vector coverage matches expected scopes.
Acceptance evidence:
- Files changed listed.
- Command output showing index/re-embed/sync-cache verification.
- Counts for files/search_fts/vec_meta/vec_chunks after verification.
- Confirmation no raw data/ vector expansion was introduced.
Activity
done
INTERACTIVE
Luci is working...
Details —
Done
· Medium
· Luci
▼
SState
Done
Closed
PPeople
TTiming / Details▼
api (human)
PKA
11d ago
11d ago
Advanced / Operator evidence
RRouting owner
ROperator console
Ticket is done; runtime is closed.ticket_runtime_doneprofile claude_opus_1m_high · cwd /home/lucienne/workspace/PKA · uptime 11d 3h · last activity 10d 20h agoMC is visibility-only. Hermes Luci launches and gates work outside MC, then mirrors evidence/status here.Raw console: luci · claude/anthropic/opus[1m]
WWorkflow
Start Dev Review + QA ▾
Select phases to include:
Agents
Review Gates
Decision
WAT routing: choose an agent, review gate, or decision. Buttons use the live runtime when one is attached.
Controller triage: decision guidance is clear (OpenAI text-embedding-3-small, portable cache by chunk_hash, exclude data/). All 6 requirements specified with acceptance evidence. Moving to todo for Luci dispatch.
luci-followup-sweep11d ago
[follow-up] 11 min past expected check-in (2026-06-02T07:50:58.802040+00:00 UTC).
luci11d ago
Luci picking up this ticket (background worker).
luci11d ago
[failed_to_inject] runtime_busy: Ticket runtime is not ready for input (status=running). Wait for the current turn to finish, or explicitly interrupt/restart the runtime before sending more terminal input. This message
Ticket picked up by Luci via MC dispatcher.
MC-4585: Vault vector embeddings: portable cache and cheaper API backend
Work this ticket in the live tmux runtime. Use DONE:, REVIEW:, or QUESTION: when you need MC to reflect the next state.
luci11d ago
[failed_to_inject] runtime_busy: Ticket runtime is not ready for input (status=running). Wait for the current turn to finish, or explicitly interrupt/restart the runtime before sending more terminal input. This message
Ticket picked up by Luci via MC dispatcher.
MC-4585: Vault vector embeddings: portable cache and cheaper API backend
Work this ticket in the live tmux runtime. Use DONE:, REVIEW:, or QUESTION: when you need MC to reflect the next state.
luci11d ago
Ticket runtime send failed while dispatching.
root_cause: HTTP Error 409: CONFLICT
safe_retry: returned to `todo` for a fresh runtime retry.
human_input_required: no
system11d ago
RUNTIME TERMINAL STATE (MC-3482 contract)
status: warning
summary: Ticket runtime send failed; parked for automatic recovery.
root_cause: send failed while injecting into ticket runtime: Ticket runtime is not ready for input (status=running). Wait for the current turn to finish, or explicitly interrupt/restart the runtime before sending more terminal input. This message was not sent.
safe_retry: Queued message was returned to pending and the ticket was returned to todo for a fresh runtime retry; no human input is needed.
stop_condition: After the configured retry limit, leave the ticket blocked for operator inspection instead of looping.
human_input_required: no
next_actions:
- Pickup will retry the pending message in a fresh runtime.
- Inspect runtime/send only if the retry fails again.
artifacts:
- ticket:MC-4585
Workflow action `review_ready` applied. Work ready for review.
luci11d ago
Shadow reviewer: ABSTAIN (advisory, low confidence). Not gating this ticket — a committed change is pinned (done_sha) but its diff is not resolvable in the known repos (slot-branch / cross-repo / reaped worktree). Orchestrator gate is authoritative.
api11d ago
Controller sign-off: VERIFIED done. Commit 61cc88ea on PKA master. Live: scripts/embed_cache.py (portable content-addressed cache keyed chunk_hash+model+dim, not path) + scripts/vec_embed.py opt-in OpenAI text-embedding-3-small@768 backend. Council-reviewed, 2 critical fixes applied, dry-run=0 pending across all scopes. Closing.
Controller decision
Luci11d ago
Controller triage: this was a technical no-live-worker/runtime failure, not a human input blocker. Cleared needs_input and returned to todo for Luci-owned execution; the broader stuck-alert/controller-action gap is tracked in MC-4593.
Luci11d ago
Controller repaired pickup 409 row drift: live tmux mc-MC-4585 exists with pane pid 1419219; normalized ticket/runtime back to in_progress.