llama-toolcall-proxy — setup

What it's for

You have a model behind an OpenAI-compatible endpoint (llama-server, most often) whose tool-call parser leaks structured calls as literal text instead of a real tool_calls field. Two shapes confirmed so far:

<tool_call>
<function=get_events>
<parameter=range>
week
</parameter>
</function>
</tool_call>

and a degenerate variant with no opening <function=> tag and a dangling unmatched </function>:

<tool_call> HassTurnOn
<parameter=name>
Kitchen Island
</parameter>
</function>
</tool_call>

This happens with brand-new model architectures running on the model author's own llama.cpp fork, before mainline support (and its parser hardening) lands. This proxy sits in front of that server and rewrites either leak shape into a proper response.

Requirements

Python 3 stdlib only. No pip install needed.

Run it

UPSTREAM_URL=http://<your-inference-server>:<port> \
LISTEN_PORT=8091 \
python3 proxy.py

UPSTREAM_URL is the server's root, without /v1 -- the proxy forwards whatever path the client requests (e.g. /v1/chat/completions) straight onto that root, so nothing about the request path changes.

Point your client at it

Wherever you'd normally set base_url = http://<server>:<port>/v1 (Open WebUI connection, an eval harness, an SDK client), swap in http://<this-proxy-host>:8091/v1 instead. Everything else -- model name, API key, request shape -- stays the same.

Verify it's working

curl http://<proxy-host>:8091/v1/models

Should return the same model list as your real server (the proxy passes GET requests straight through). Then send a real tool-calling chat request and confirm the leaked-text failure is gone and tool_calls is populated.

Tests

python3 -m pytest test_proxy.py -q

19 tests, all against real leaked-text samples pulled from live evaluations (see the boxxo-model-bench repo's FINDINGS.md and FINDINGS_HA_ASSIST.md for where they came from).

Proof it works

Two independent evals, two different tool-schema sizes, both via boxxo-model-bench's harness:

	Direct	Through proxy
Boxxo (44-tool schema)	9/27 (33%)	19/27 (70%)
HA Assist (13-tool schema)	11/14 (79%)	12/14 (86%)

Don't assume a small tool schema is safe to skip this for -- the leak showed up at 13 tools too (in the second shape above), just less often than at 44. It's probability-correlated with schema size, not a hard cliff you can rule out below some threshold.

When to stop using this

Once the model's tool-call support lands in mainline llama.cpp (or your inference stack's own parser gets fixed for your tool-schema scale), this proxy becomes unnecessary -- just point your client back at the real server directly.