grimoire-toolkit

Companion automation for Grimoire — a self-hosted TTRPG library and campaign manager. This toolkit fills gaps around Grimoire's own features (OCR, metadata, library scanning) rather than replacing them: one-time maintenance fixes, metadata enrichment pulled from itch.io/DriveThruRPG and AI-assisted extraction, and a drop-folder watch service for new arrivals.

Talks to Grimoire only through its REST API and — where the API has no equivalent — direct, read-verified database access, using a dedicated automation service account. It never modifies Grimoire's own code.

Status

Early — built incrementally, stage by stage. See docs/superpowers/specs/ for the design of each piece as it's built.

Stage A — scan_failed fix

fix_scan_failed.py fixes a Grimoire indexer bug: some books get permanently flagged scan_failed=1 even though they were indexed/thumbnailed successfully, and Grimoire's own scanner never retries a row already flagged this way. The script verifies each row actually succeeded before clearing the flag — anything that doesn't pass gets reported, not touched.

Run it inside the Grimoire container (see the script's own docstring for the exact commands), then trigger a rescan via GrimoireClient().trigger_rescan() so Grimoire backfills the thumbnail/page-count fields for any real PDFs that were affected.

Setup

pip install -r requirements.txt
cp .env.example .env   # fill in GRIMOIRE_AUTOMATION_PASSWORD

For local development, pip install -e . (uses pyproject.toml's [project] metadata) installs this repo's own packages (itch_pipeline, ai_extraction, review_webui) plus its direct dependencies in one step, and adds a grimoire-toolkit console script equivalent to python3 cli.py. The Dockerfile still installs from requirements.lock.txt directly (no editable install inside the container) and every deployed mode still invokes python3 cli.py <mode> — this is a dev convenience, not a change to how the container runs.

requirements.txt is the direct/hand-edited dependency list (unpinned to a specific patch of the dependency tree). requirements.lock.txt is a full transitive freeze the Dockerfile actually installs from, for a reproducible container build. After changing requirements.txt, regenerate it:

python3 -m venv /tmp/lockenv && /tmp/lockenv/bin/pip install -r requirements.txt \
  && /tmp/lockenv/bin/pip freeze > requirements.lock.txt

Stage 0 — itch.io fetch + TTRPG system detection

Ported from the standalone itch-library-pipeline tool: cli.py fetch downloads owned itch.io books; cli.py system detects which shared TTRPG system an existing library's folders belong to (itch.io tags, PDF license-phrase text, title-based clustering for core+expansion pairs) and reorganizes them into Grimoire's books/<system>/core/<title>/ layout. Dry-run by default — writes system_plan.csv/system_discovery.csv for review; add --commit to actually move files.

The detection logic is source-pluggable (itch_pipeline/sources/) — ItchSource and DtrpgSource (DriveThruRPG) both run today; RPGGeek is deferred (see Stage 0.5 below).

This is now the sole home for this logic — the original standalone itch-library-pipeline repo is archived (see ../archived-itch-library-pipeline). Its BookLore/Komga-specific modes (sort/report/apply/watch/series/comics) were not ported; they targeted a different app family and are out of scope here.

Note on validation: the port was validated via 61/61 passing tests (every ported function independently verified byte-identical to the original upstream logic during code review, task by task) plus a live network dry-run smoke test. The original plan called for also diffing a fresh dry-run against the real library against the old tool's output on the same library, but that library had already been reorganized (and its unorganized original deleted) by an earlier, already-completed real migration — so that specific comparison could no longer be run. Worth being aware of if a subtle detection regression ever turns up that the test suite didn't catch.

Usage

python3 cli.py fetch --staging /path/to/staging
python3 cli.py system --src-dir /path/to/existing/library --dst-dir /path/to/grimoire/books
# review system_plan.csv, then:
python3 cli.py system --src-dir /path/to/existing/library --dst-dir /path/to/grimoire/books --commit

Stage 0.5 — DriveThruRPG source + deeper PDF sampling

system mode now also checks DriveThruRPG's curated "Rule System" filter taxonomy (a product-detail API endpoint, no auth needed) alongside itch.io tags — this catches traditional/major-publisher games itch.io never carried at all. pdf_text_sample()'s sampled page range was also widened (3/2 → 5/3 front/back pages) so a license notice sitting a bit further into a document is less likely to be missed.

RPGGeek was originally planned as a third source but is deferred: BoardGameGeek's XML API (RPGGeek shares its infrastructure, not a separate host) now requires registering an application at boardgamegeek.com/applications and waiting for manual approval (documented turnaround: "a week or more") to get a Bearer token — there's no unauthenticated path anymore. This is an external, human step; once a token exists, RpggeekSource is a self-contained follow-up (same Source interface).

Validated with live network requests against the real DriveThruRPG API (not just mocked fixtures) — DtrpgSource().search("Cairn") and .tags_for() on the winning result correctly returned ["Cairn"] end to end.

Stage B — AI extraction engine + loose-file classification

For files with no folder-name signal at all to detect from -- a flat unsorted/ drop folder of loose scans, many with no extractable text layer whatsoever -- cli.py extract-loose uses a vision model to read the title/author/system straight off the page images, then organizes the result through the exact same search-verification and confidence-tiering pipeline system mode already uses (candidate_ranking.py's best_match()/tier(), unchanged).

Local-first: qwen3.5:9b (Ollama, on this household's GPU PC) is tried first, on a narrow front/back page sample, widening the sample once if that doesn't yield a usable guess. If the local model still can't answer, or the tier comes back MEDIUM (needs a second opinion), it escalates to a cloud chain -- Gemini (which can read the raw PDF directly), then Groq, Mistral, and OpenRouter's free tier, in that order, stopping at the first usable result.

A system the vision model reads confidently but that isn't yet in the curated KNOWN_SYSTEMS dict still gets its own correctly-named, correctly-clustered folder (see docs/superpowers/specs/2026-07-15-stage-b-ai-extraction-engine.md for exactly how) -- and gets reported in new_system_candidates.csv so it can be promoted into the curated dict by hand later.

Matches system mode's dry-run-by-default convention: every run writes extract_report.csv (and new_system_candidates.csv) without moving anything; add --commit to actually move HIGH-tier results. Anything that doesn't reach HIGH confidence always lands in the report for manual review, --commit or not.

Usage

python3 cli.py extract-loose --src-dir /path/to/unsorted --dst-dir /path/to/grimoire/books
# review extract_report.csv and new_system_candidates.csv, then:
python3 cli.py extract-loose --src-dir /path/to/unsorted --dst-dir /path/to/grimoire/books --commit

Manual-review companion page

For whatever extract-loose couldn't confidently organize on its own, cli.py review starts a small local web page (styled to match Grimoire's own look) listing every pending-review file from a prior run's extract_report.csv. Click into one, look at the same page images the extraction engine already sampled, correct the title/author/system, and hit "Check" -- it re-runs the exact same search-verification extract-loose itself uses. Once a correction reaches HIGH confidence, "Confirm move" organizes the file on the spot, re-verifying fresh at that exact moment rather than trusting whatever an earlier check said (so editing the fields again after checking, before confirming, can never move a file based on stale data).

Usage

python3 cli.py review --outdir /path/to/extract-loose/outdir --dst-dir /path/to/grimoire/books
# opens on http://127.0.0.1:5000 by default; add --port to change it

Stage C1 — Metadata enrichment

cli.py enrich fills in blank authors/publisher/publisher_url/description/year fields on books already organized into Grimoire, using the same search-verification pipeline (HIGH tier only) system/extract-loose already use. Only a book's blank fields ever get written -- anything already filled in is left alone, even if a fresh search match disagrees with it. A disagreement (or a book with no confident match at all, or a search match whose own curated system doesn't match the book's current folder) is never silently applied -- it lands in enrich_review_queue.csv for a human to look at instead. Re-filing a book under a different system isn't something this stage can do at all -- Grimoire's own API silently ignores an attempt to change game_system_id, confirmed live during design -- so a system mismatch is flagged, not fixed.

--mode missing (default) only processes books with at least one blank target field; --mode full re-checks every book regardless of current state (useful for an initial pass, or a deliberate full re-verification later).

Dry-run by default, matching every other mutating mode in this toolkit -- pass --commit to actually call Grimoire's API. enrich_report.csv's action column reads would-fill on a dry-run and filled once committed, so a dry-run report previews exactly what a real run would do. (The webui's /enrich/start scan trigger, Stage C2 below, always commits -- there's no separate preview step there.)

A webui review/correct page over enrich_review_queue.csv is Stage C2, below.

Usage

python3 cli.py enrich --mode missing --outdir /path/to/enrich-out
# dry-run: review enrich_report.csv (what WOULD be auto-filled) and enrich_review_queue.csv
python3 cli.py enrich --mode missing --outdir /path/to/enrich-out --commit
# actually fills the fields

Stage C2 — Enrichment review webui

cli.py review's companion page (see "Manual-review companion page" above) gains a second section: visit /enrich to trigger an enrich scan from the browser (missing or full mode) and watch its live progress, without freezing the rest of the page -- the scan runs in a background thread, so the existing extract-loose review flow stays usable while it's going. A "Sync library now" button on the same page triggers Grimoire's own rescan and waits for it to actually finish before cleaning up now-missing records -- calling those two Grimoire endpoints back-to-back without waiting fails, since Grimoire's own cleanup rejects while a scan is still running.

/enrich/queue renders enrich_review_queue.csv grouped by what kind of attention it needs:

• Field conflicts (an already-filled field a fresh search disagrees with) -- apply the proposed value or keep the current one, one click either way.
• No confident match -- correct the title/author and recheck (same fresh search-verification the rest of this toolkit already uses); a HIGH result offers to apply the proposed fields.
• System mismatches (a book's own tags suggest it's filed under the wrong system) -- move the file into the correct system folder. Grimoire itself has no way to re-link a moved file to its existing database record (confirmed from its own source: books are matched purely by exact file path, no content-based re-linking) -- so the moved book simply re-enters this same enrichment pipeline fresh on the next scan, rather than trying to carry its old metadata forward through a fragile lookup.

Usage

python3 cli.py review --outdir /path/to/enrich-out --dst-dir /path/to/grimoire/books
# opens on http://127.0.0.1:5000 by default -- visit /enrich for the scan trigger + review queue

---

Built with the help of Claude Code (Claude Sonnet 5).