Setup & Usage Guide

What this solves

If your Karakeep deployment wants embeddings on one Ollama instance and tagging/summarization on a different one (e.g. a small dedicated GPU for embeddings, a bigger shared GPU for tagging), this proxy is the glue. Karakeep only supports one OLLAMA_BASE_URL, so this sits in front of it and routes each request to the right backend based on which model is being asked for.

Step 1 — Build the image

git clone https://git.ourserver.party/xenarathon/karakeep-ollama-router.git
cd karakeep-ollama-router
docker build -t karakeep-ollama-router:latest .

No build args needed — it's a plain python:3.13-slim base with one script copied in.

Step 2 — Run it alongside your two Ollama backends

Add it to your compose file (same project/network as Karakeep so they can reach each other by service name):

  karakeep-router:
    image: karakeep-ollama-router:latest
    environment:
      EMBED_UPSTREAM: http://ollama-embed:11434
      TAG_UPSTREAM: http://192.168.1.196:11434
      EMBED_MODEL: mxbai-embed-large
      LISTEN_PORT: "8000"

EMBED_UPSTREAM and TAG_UPSTREAM are just base URLs of any two real Ollama servers — they don't need to be special in any way, they just need to actually have the models you're routing to them.

EMBED_MODEL must match the exact model name Karakeep sends for embeddings (i.e. whatever you set EMBEDDING_TEXT_MODEL to in Karakeep's own config). Anything else falls through to TAG_UPSTREAM.

Step 3 — Point Karakeep at the proxy

In Karakeep's own environment:

OLLAMA_BASE_URL: http://karakeep-router:8000

That's it — Karakeep has no idea it's talking to a proxy, and no code changes are needed on its side.

Sanity-checking it works

From inside any container on the same network as the router:

# should hit EMBED_UPSTREAM
curl -s http://karakeep-router:8000/api/embeddings \
  -d '{"model":"mxbai-embed-large","prompt":"hello"}'

# should hit TAG_UPSTREAM
curl -s http://karakeep-router:8000/api/generate \
  -d '{"model":"your-tagging-model","prompt":"hello","stream":false}'

Logging

The router prints startup info (its configured upstreams) to stdout on boot — check docker logs <container> if routing looks wrong. It doesn't log per-request decisions by default — add a print() in _pick_upstream if you need to debug routing decisions for a specific request.

Gotchas

• No host port needed. The router only needs to be reachable by Karakeep itself, so a ports: mapping is optional — add one only if you want to curl it directly from the host for debugging.
• It doesn't validate model names. If you typo EMBED_MODEL so it never matches what Karakeep actually sends, embedding requests will silently go to TAG_UPSTREAM instead — if that server also happens to have the same model pulled, you won't notice anything is wrong until you check which backend actually served it. Verify with a real log check on both backends the first time.
• HTTP/1.0 on purpose. Responses are streamed through without buffering, so the proxy can't always set an accurate Content-Length up front; it uses connection-close semantics instead. If you modify the code to add HTTP/1.1 keep-alive, make sure you keep Content-Length/chunked-encoding correct or clients will hang.

Real-world example

Deployed for a household Karakeep instance: embeddings route to a dedicated 2GB card running only mxbai-embed-large (kept permanently resident with OLLAMA_KEEP_ALIVE=-1), while tagging/ summarization routes to a shared 8GB card running a larger model on demand.