xenarathon/boxxo-model-bench
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Comparative A/B test harness for swapping the backend LLM behind a persona-driven, tool-calling assistant
8 commits 1 branch 0 tags 108 KiB
Find a file
Exact
xenarathon 9b928cfe30 Add findings for remaining consumers: Boxxo Router, Nyarch, digarr 
Closes out the consumer sweep. Boxxo Router fast-path confirmed
redundant with the existing Boxxo eval (same model/endpoint/tool
shape). Nyarch Assistant is closed-source, not testable. digarr tested
directly (open-source prompt) and found a real capability gap: gets
stuck in a repetition loop on long, open-ended list generation with a
uniqueness constraint, unlike its solid tool-calling/bounded-JSON
performance in the other three evals.
2026-07-24 13:27:11 -04:00
.gitignore Add second eval: HA Assist voice pipeline, qwen3:8b vs Nanbeige4.2-3B 2026-07-24 11:33:53 -04:00
example.ha_system_prompt.txt Add second eval: HA Assist voice pipeline, qwen3:8b vs Nanbeige4.2-3B 2026-07-24 11:33:53 -04:00
example.system_prompt.txt Add boxxo-model-bench: comparative A/B harness for swapping backend LLMs 2026-07-23 17:29:17 -04:00
FINDINGS.md Update findings: proxy patch confirms parser-not-model hypothesis 2026-07-24 10:08:55 -04:00
FINDINGS_HA_ASSIST.md Add second eval: HA Assist voice pipeline, qwen3:8b vs Nanbeige4.2-3B 2026-07-24 11:33:53 -04:00
FINDINGS_JSON_TASKS.md Add third eval: structured JSON tasks, qwen3:8b vs Nanbeige4.2-3B 2026-07-24 13:15:12 -04:00
FINDINGS_OTHER_CONSUMERS.md Add findings for remaining consumers: Boxxo Router, Nyarch, digarr 2026-07-24 13:27:11 -04:00
harness.py Add configurable per-target max_tokens 2026-07-23 21:30:29 -04:00
harness_json.py Add third eval: structured JSON tasks, qwen3:8b vs Nanbeige4.2-3B 2026-07-24 13:15:12 -04:00
LICENSE Add boxxo-model-bench: comparative A/B harness for swapping backend LLMs 2026-07-23 17:29:17 -04:00
README.md Add second eval: HA Assist voice pipeline, qwen3:8b vs Nanbeige4.2-3B 2026-07-24 11:33:53 -04:00
REVIEW.md Add boxxo-model-bench: comparative A/B harness for swapping backend LLMs 2026-07-23 17:29:17 -04:00
test_harness.py Add boxxo-model-bench: comparative A/B harness for swapping backend LLMs 2026-07-23 17:29:17 -04:00
tests.json Add boxxo-model-bench: comparative A/B harness for swapping backend LLMs 2026-07-23 17:29:17 -04:00
tests_ha_assist.json Add second eval: HA Assist voice pipeline, qwen3:8b vs Nanbeige4.2-3B 2026-07-24 11:33:53 -04:00
tests_json_tasks.json Add third eval: structured JSON tasks, qwen3:8b vs Nanbeige4.2-3B 2026-07-24 13:15:12 -04:00
tools_ha_assist.json Add second eval: HA Assist voice pipeline, qwen3:8b vs Nanbeige4.2-3B 2026-07-24 11:33:53 -04:00

README.md

boxxo-model-bench

A comparative A/B test harness for swapping the backend LLM behind a persona-driven, tool-calling assistant. Built for Boxxo, a household homelab assistant, but the harness itself doesn't know anything about Boxxo specifically — point it at any two OpenAI-compatible chat endpoints, any system prompt, and any MCP tool server, and it'll score them against the same test suite side by side.

Why this exists

New small/local models show up constantly claiming to beat bigger incumbents on public benchmarks. Benchmark numbers don't tell you whether a model reliably emits real structured tool calls for your actual tool set, keeps your persona's tone, or handles your specific edge cases (ambiguous targets, no-guessing rules, multi-turn corrections). This harness runs the same test suite against a baseline (your current production model) and a candidate (the new thing you're evaluating) and reports a real, apples-to-apples comparative score — never a new run compared against some old, different suite's historical number.

What it checks

Four dimensions, per test case:

  • Personality — does the response carry the persona's expected tone/markers, without contradictory stacking (e.g. a failure response that also claims success)?
  • Tool selection — does it call the correct tool (checked against the real, live tool schemas from your MCP server), and avoid calling the wrong one?
  • Data relay — given a mocked tool result, does the final answer surface the actual values, not a vague "I found it"?
  • Hard edge cases — no-guessing discipline, relaying ambiguous-match options instead of picking or refusing, honest failure reporting instead of claiming "I don't have access", and multi-turn correction handling.

Tool results are always mocked, never real — a test suite should never actually turn off your lights or post an announcement to your family.

Design notes
  • Both models are hit directly over their own /v1/chat/completions endpoint (Ollama, llama-server, vLLM — anything OpenAI-compatible), not through whatever chat UI sits in front of them in production. This keeps runs hermetic and avoids that UI's own quirks (registry flakiness, caching, prompt injection) muddying the model-vs-model comparison. Do a small manual sanity pass through your real UI afterward for whichever model wins.
  • Config is entirely environment-driven so this repo stays generic and reusable for the next model you want to A/B, instead of being rewritten each time.
  • Tool schemas are fetched live from a running MCP server by default, so the suite is always graded against what's actually deployed, not a stale copy.
  • If both models can't fit in VRAM at once, use ONLY=baseline/ONLY=candidate with SAVE_RESULTS/LOAD_BASELINE_RESULTS to run them sequentially and still get a full comparative diff.
  • A low score is not evidence of a bad model until you've read the raw transcripts for a sample of the failures. Two harness misconfigurations produced misleadingly low scores during this tool's first real evaluation, both of which looked exactly like genuine model failures until the transcripts were inspected. See FINDINGS.md for the full writeup — it's as much a case study in how to not trust a benchmark number as it is a model evaluation.

See FINDINGS.md for a worked example evaluation (Nanbeige4.2-3B vs qwen3.5:9b, Boxxo) and FINDINGS_HA_ASSIST.md for a second, independent one (same candidate vs. qwen3:8b, Home Assistant's voice assistant) — together they're evidence the harness generalizes across consumers, not just its original use case. Setup and usage details are in the wiki.

License

GPL-3.0-or-later — see LICENSE.

Credit

Built with Claude Code.