Setup & Usage

Requirements

• Python 3, standard library only — no pip install needed.
• Two model endpoints, each exposing an OpenAI-compatible /v1/chat/completions route (Ollama, llama-server, vLLM, etc. all work).
• A tool schema for whatever domain you're testing. Either:
  • an MCP server (streamable-http) to fetch live tool schemas from, or
  • a hand-written TOOLS_FILE in the same array-of-objects shape MCP's tools/list returns (name, description, inputSchema) — useful when there's no MCP server, e.g. reconstructed from a framework's own source code (see tools_ha_assist.json for a worked example: Home Assistant's built-in Assist intent tools, sourced from HA's own repo rather than live-captured).

The harness doesn't know or care which domain it's testing — swap in a different SYSTEM_PROMPT_FILE / TOOLS_FILE / TESTS_FILE triple per target. Two examples ship in this repo: system_prompt.txt + mcp_tools.json + tests.json (Boxxo, a persona-driven assistant with 44 custom MCP tools) and ha_system_prompt.txt (gitignored — see example.ha_system_prompt.txt) + tools_ha_assist.json + tests_ha_assist.json (Home Assistant's voice pipeline, 13 built-in intent tools).

Quick start (both models resident at once)

BASELINE_URL="http://192.168.1.10:11434/v1" \
BASELINE_MODEL="qwen3.5:9b" \
CANDIDATE_URL="http://192.168.1.20:8080/v1" \
CANDIDATE_MODEL="/models/candidate-Q8_0.gguf" \
SYSTEM_PROMPT_FILE="./system_prompt.txt" \
MCP_URL="http://192.168.1.10:8100/mcp" \
TOOLS_FILE="./mcp_tools.json" \
TESTS_FILE="./tests.json" \
python3 harness.py

SYSTEM_PROMPT_FILE should point at your assistant's real persona/system prompt — a plain text file, not committed to this repo (see example.system_prompt.txt / example.ha_system_prompt.txt for generic stand-ins and to understand the shape expected).

Always validate config first with DRY_RUN=1 — it loads everything (tools, tests, prompt) and prints the test plan without calling any model:

DRY_RUN=1 <same env vars> python3 harness.py

Single shared GPU (models can't both be resident in VRAM)

Run each side separately, freeing VRAM in between:

# 1. Make sure only the baseline's server is running (stop the candidate's).
ONLY=baseline \
BASELINE_URL=... BASELINE_MODEL=... \
SYSTEM_PROMPT_FILE=./system_prompt.txt MCP_URL=... TOOLS_FILE=./mcp_tools.json TESTS_FILE=./tests.json \
SAVE_RESULTS=./results/baseline_latest.json \
python3 harness.py

# 2. Stop the baseline's server / unload the model, start the candidate's server.
#    (If the candidate's inference stack leaks tool calls as text instead of
#    structured tool_calls, see llama-toolcall-proxy -- put it in front of
#    the candidate's server and point CANDIDATE_URL at the proxy instead.)

# 3. Run the candidate side, diffed against the saved baseline.
ONLY=candidate \
CANDIDATE_URL=... CANDIDATE_MODEL=... \
SYSTEM_PROMPT_FILE=./system_prompt.txt MCP_URL=... TOOLS_FILE=./mcp_tools.json TESTS_FILE=./tests.json \
LOAD_BASELINE_RESULTS=./results/baseline_latest.json \
python3 harness.py

The second run prints the same side-by-side comparative summary as a single-process run would.

If a model reasons even when you ask it not to

Some models/backends don't honor a think-suppression flag over their OpenAI-compat endpoint and burn through the token budget on reasoning before producing any real content — this reads as a timeout, not a wrong answer. If you see a wall of exception: timed out failures, check one raw response for a reasoning/reasoning_content field consuming the whole max_tokens budget, then raise BASELINE_MAX_TOKENS/CANDIDATE_MAX_TOKENS (and the matching _TIMEOUT) until it actually finishes a real answer.

Writing test cases

See tests.json (or tests_ha_assist.json for a second worked example) for the full set and the module docstring in harness.py for the field reference. The short version: each case sends a user message, optionally checks which tool got called (expect_tool, forbidden_tools), optionally feeds back a mock_tool_result and checks the final answer against response_regex/forbidden_regex, and optionally adds a follow_up user turn to test multi-turn correction. Tool results are always mocked — this harness should never be pointed at a live tool server that can actually change state.

Careful with follow_up + ambiguous scenarios: if the model reasonably asks a clarifying question instead of calling a tool on the first turn (e.g. a genuinely ambiguous target), the mocked-result/follow-up half of the test never executes, since that logic only fires when a tool was actually called. A case can "fail" this way for a completely reasonable response — check the transcript before assuming it's a real capability gap (see FINDINGS_HA_ASSIST.md's multiturn-correction case for a worked example of exactly this).

Before trusting a low score

Read the raw transcript for a handful of the failures before concluding the candidate model is actually worse. See FINDINGS.md (Boxxo, 44 tools) and FINDINGS_HA_ASSIST.md (Home Assistant, 13 tools) in the repo root for two real examples where harness misconfigurations or test-design assumptions produced misleadingly low scores that looked exactly like genuine model failures.