vault backup: 2026-08-12 15:38:55

zz-MAIN/RUNBOOK.md

@@ -0,0 +1,116 @@
+# RUNBOOK — ky_court_pipeline.ipynb
+
+One notebook to restore, discover, download, extract, sync, and verify the full
+Kentucky appellate court document corpus (~98.7k documents). It is designed to
+run in Google Colab, auto-resume after disconnects, and mirror all state up to a
+Drive folder so nothing is lost across sessions.
+
+## What this is
+
+A single generated notebook (`ky_court_pipeline.ipynb`) whose cells are inlined
+from real Python modules in `notebook_builder/cells/`. The notebook is rebuilt
+with `notebook_builder/build.py`. All stages read/write the same SQLite DB and a
+`Store` abstraction that is Google Drive in Colab and the local filesystem in
+tests/smoke.
+
+Stage order: **restore → discover → backfill → download → extract → sync/verify**.
+Any stage stops gracefully at the session budget; re-running resumes where it
+left off.
+
+## Files on Drive (`MyDrive/ky_court-WIP/`)
+
+| Path | Purpose |
+|---|---|
+| `ky_court.sqlite` | source of truth (documents, discovery_runs, kv, extraction_log) |
+| `pdf_by_hash/<sha256>.pdf` | deduped PDFs keyed by content hash (3,326 unique from 9,582 backup files) |
+| `pdf_backup/<case>_<n>.pdf` | original named backups — **retained, never deleted** |
+| `text/<sha256>.txt` | extracted text per PDF |
+| `exports/file_manifest.csv` | sha256 → size manifest |
+| `exports/document_inventory.csv` / `failed_inventory.csv` | exported summaries from verify |
+| `status.json` | live heartbeat + phase/counts (the "is it alive" check) |
+| `ky_court-WIP-resume-*.tar.gz` | (MyDrive root) resume tarball used to rehydrate a fresh runtime |
+
+The store root is `MyDrive/ky_court-WIP` (Colab) or `<WORKDIR>/ky_court-WIP` (local).
+
+## First run
+
+1. If Drive is empty (no `ky_court-WIP`), upload `ky_court-WIP-resume-YYYY-MM-DD.tar.gz`
+   to `MyDrive/`. Stage **restore** will extract it.
+2. Open `ky_court_pipeline.ipynb` in Colab.
+3. Edit the first code cell (`CFG`) if needed (budget, phases — see Config knobs).
+4. **Runtime > Run all**. A session mounts Drive, opens the DB (pulling it from
+   the store if absent), heals stale state, then runs the enabled phases.
+
+Expected session timeline (typical):
+- setup: ~1 min (Drive mount, dirs, recover)
+- discover: 10–30 min (74 shards, paged search, paced)
+- backfill: ~20 min (hash `pdf_backup` → dedupe into `pdf_by_hash`)
+- download: ~10 h per session ≈ 6–12k documents (budget-capped)
+- extract: rides along between batches (thread-pooled PDF text extraction)
+- verify: ~5 min (integrity check, inventory exports, status push)
+
+## Resuming
+
+Just re-run the notebook. `recover()` closes any stale `running` discovery runs,
+drops leftover `.part` files, and each stage skips what's already done:
+- backfill: guarded by `kv backfill_done`
+- download: only `discovered`/`failed` docs with `attempts < MAX_ATTEMPTS`
+- extract: only `downloaded` docs with no extracted text yet
+
+Idempotent by design — safe to rerun as often as you like.
+
+## What "done" looks like
+
+Verify prints `PASS {checks}` and, when every queue is empty:
+- notebook verify output includes `done: True`
+- `status.json` on Drive contains `"done": true`
+
+A session that merely hit its budget is **not** "done" — it stops early with
+`status.json` showing the phase and remaining queues. That is normal mid-flight
+state, not an error.
+
+## Failure modes
+
+- **Colab disconnect mid-run** → just re-run. `recover()` heals stale rows;
+  `.part` files are dropped; partial downloads retry from the queue.
+- **Drive quota exceeded** → verify/checkpoint push fails; the run continues
+  locally in the session but the store lags. Check Drive quota, free space, re-run.
+- **5xx / connection storms** → `api_download` returns an error tuple and
+  `stage_download` backs off `min(60, 5*2^attempts)` seconds, retrying up to
+  `MAX_ATTEMPTS` before marking the doc `failed`.
+- **Search shard hits the 10k ceiling** → `api_search_pages` raises; the planner
+  sub-shards that seed (`YYYY-CA-N`) on the next run.
+- **`integrity` check not "ok"** → verify prints `FAIL`; the DB is corrupted and
+  should be restored from the last resume tar before trusting anything.
+
+## Config knobs (top code cell `CFG`)
+
+| Key | Default | Meaning |
+|---|---|---|
+| `BUDGET_SECONDS` | 10 * 3600 | max wall-clock per session (env `KY_BUDGET`) |
+| `SHUTDOWN_SECONDS` | 900 | stop stages this long before the deadline so the session can finish cleanly |
+| `PACING_SEARCH` / `PACING_DL` | (0.3,0.7) / (0.3,0.5) | random sleep range between API calls (polite throttling) |
+| `MAX_ATTEMPTS` | 4 | download retries before `failed` |
+| `MAX_PDF_BYTES` | 200 MiB | size cap; oversized (HTTP 413) fails permanently, no retry/backoff |
+| `MAX_EXTRACT_ATTEMPTS` | 3 | PDF-extraction retries before skip |
+| `RETRY_HARD` | 0 (env `KY_RETRY_HARD`) | on resume, reset `failed` docs back to `discovered` |
+| `PHASES` | discover,backfill,download,extract | comma-separated phases to run (env `KY_PHASES`) |
+| `DRY_RUN` | 0 (env `KY_DRY`) | 1 = fully offline: discovery/download skip all network |
+
+## Safety notes
+
+- The pipeline **never deletes on Drive automatically**. `pdf_backup` is retained;
+  pre-restore move-aside only renames (`-pre-restore`), never deletes.
+- Set `DRY_RUN=1` (`KY_DRY=1`) to exercise the whole flow offline with zero
+  network calls — the smoke script (`scripts/smoke_local.py`) does exactly this.
+- Live API tests are gated behind `KY_LIVE=1` and share a ≤10-request budget;
+  keep them there so the ordinary test suite stays offline and fast.
+
+## Local / tests
+
+```bash
+python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
+.venv/bin/pytest tests -q               # 18 offline tests (live ones skipped)
+.venv/bin/python scripts/smoke_local.py # full pipeline over the real 98,724-row DB
+.venv/bin/python notebook_builder/build.py  # regenerate the notebook
+```