MC-4861 — Benchmark local coding models on Lucienne MacBook

Home Board MC-4861

Benchmark local coding models on Lucienne MacBook

Follow-up to MC-4845. Run the same local coding/grunt-work model benchmark on the MacBook Lucienne: verify hardware, runner availability (LM Studio/lms and/or llama.cpp), practi...

State Done Next Action Closed Owner Lucienne Runtime Closed Age 4d ago

← MC-4861

Ticket is done; runtime is closed. · profile claude_opus_1m_medium

Description

MC-4861

Follow-up to MC-4845. Run the same local coding/grunt-work model benchmark on the MacBook Lucienne: verify hardware, runner availability (LM Studio/lms and/or llama.cpp), practical GGUF/model candidates already present or downloadable, OpenAI-compatible endpoint path for Hermes, resource behavior, and produce concrete recommendation with tested commands/endpoints.

Activity

done

Luci is working...

Details — Done · High · Lucienne ▼

State

Done

Closed

State

Priority

People

Owner (assigned to)

Controller

Timing / Details

Due Date

Snooze

Source api (human)

Project General

Created 4d ago

Updated 4d ago

Advanced / Operator evidence

Routing owner

Routes via

Operator console

Evidence

Ticket is done; runtime is closed. MC is visibility-only. Hermes Luci launches and gates work outside MC, then mirrors evidence/status here.

Workflow

Start Dev Review + QA ▾

Select phases to include:

Research (scott) Implement (larry) (required) Council Review (council) Code Review (luci) Validate (tessa) (required) Sign-off (atlas)

Agents

Review Gates

Decision

WAT routing: choose an agent, review gate, or decision. Buttons use the live runtime when one is attached.

system 4d ago

[visibility-only] Queued message recorded for Hermes Luci. MC did not claim the ticket or spawn a runtime.

lucienne 4d ago

Starting MacBook Lucienne follow-up benchmark from MC-4845: local hardware/runner/model/resource checks plus OpenAI-compatible endpoint recommendation.

lucienne 4d ago

Completed Lucienne MacBook follow-up benchmark. Evidence: /Users/elmar/PKA/reports/mc-4861_lucienne_local_model_benchmark.md plus raw JSON/scripts in /Users/elmar/PKA/reports/. Recommendation: use LM Studio qwen/qwen2.5-coder-14b via http://127.0.0.1:1234/v1; avoid 30B/32B downloads until disk is freed (~15Gi free); qwen3.5-9b/gemma showed hidden-reasoning/blank-output issues; Ollama llama3.1:8b is fast fallback but weaker for coding/shell.

lucienne 4d ago

Follow-up: added Hermes custom provider 'lmstudio' pointing to http://127.0.0.1:1234/v1 with default model qwen/qwen2.5-coder-14b, loaded the model in LM Studio with 64K context, and verified Hermes CLI works via: hermes chat --provider lmstudio -m qwen/qwen2.5-coder-14b. Also verified a simple terminal-tool call succeeds. Telegram/gateway sessions may need restart/reset before the new provider is visible there.

lucienne 4d ago

Reopened after real Hermes/TUI evidence: previous benchmark was insufficient. Raw LM Studio endpoint tests are not enough; acceptance must include Hermes full prompt with at least one skill/tool-loaded session and >8K prompt/context. Screenshot shows qwen/qwen2.5-coder-14b failing at ~12K despite Hermes assuming 65,536 context, so current LM Studio/Qwen setup is NOT approved for Hermes agent use.

lucienne 4d ago

Correction after Elmar's Hermes screenshot: previous raw endpoint benchmark has been superseded. Added real Hermes acceptance harness at /Users/elmar/PKA/reports/mc-4861_hermes_local_model_acceptance.py and results at /Users/elmar/PKA/reports/mc-4861_hermes_local_model_acceptance.json. Current qwen/qwen2.5-coder-14b + LM Studio setup FAILS all Hermes acceptance cases: minimal, default profile, skill-loaded, terminal tool, and >8K prompt. Updated report with NOT APPROVED verdict. Removed the lm-coder model alias so it is not presented as a working shortcut; left provider config for further testing.

lucienne 4d ago

Control room tick (Jun 9): Benchmark work complete — all Hermes acceptance cases FAIL for qwen/qwen2.5-coder-14b + LM Studio (context window misreport, tool-call failures). Evidence at /Users/elmar/PKA/reports/mc-4861_hermes_local_model_acceptance.json. Verdict: NOT APPROVED for Hermes agent use. Moving to in_review for Elmar to decide next steps (try other models, accept limitation, or close).

lucienne 4d ago

Control room (Jun 9): Closing as done. Benchmark deliverable complete — comprehensive report at /Users/elmar/PKA/reports/mc-4861_lucienne_local_model_benchmark.md with acceptance harness results at mc-4861_hermes_local_model_acceptance.json. Verdict: qwen/qwen2.5-coder-14b + LM Studio NOT APPROVED for Hermes agent use (context window misreport, tool-call failures across all test cases). If Elmar wants to explore other local models or configurations, that would be a new ticket.

luci-board-manager 4d ago

Controller processed the legacy queued_message because Lucienne has already started the external MacBook benchmark in MC comments; MC visibility-only did not spawn a runtime.

luci-board-manager 4d ago

Controller hygiene: cleared stale MC-runtime pressure on this visibility-only external Lucienne task. No MC ticket runtime is expected or allowed here; the substantive handoff is the Lucienne comment that started the MacBook benchmark. Leaving the ticket in_progress with assigned_to=lucienne and no MC pickup/runtime action.

luci-board-manager 4d ago

Controller review gate: MC-4861 has a concrete Lucienne handoff for the Hermes local-model acceptance benchmark. The reported verdict is NOT APPROVED for qwen/qwen2.5-coder-14b + LM Studio because the real Hermes acceptance harness failed minimal/default/skill/terminal/>8K cases. Evidence path in the Mac handoff: /Users/elmar/PKA/reports/mc-4861_hermes_local_model_acceptance.json. Leaving this in_review as a genuine human decision CTA: choose whether to try other local models, accept the limitation, or close the benchmark as 'not approved'. No MC runtime/pickup was launched.

Live ▼

No activity yet

←