vault backup: 2026-08-12 15:38:55

2026-08-12 15:38:55 -04:00
parent 594346c495
commit 22c31d7bd4
23 changed files with 1255 additions and 155 deletions
@@ -0,0 +1,59 @@
+# ky-court-colab
+
+A single generated Google Colab notebook that **restores, discovers, downloads,
+extracts, and verifies** the full Kentucky appellate court document corpus
+(~98.7k documents). Built to survive Colab disconnects — it auto-resumes, is
+idempotent, and mirrors all state to Google Drive so nothing is lost between
+sessions.
+
+## What it does
+
+`ky_court_pipeline.ipynb` walks the corpus through six stages:
+
+```
+restore → discover → backfill → download → extract → sync/verify
+```
+
+Every stage reads/writes one SQLite DB and a `Store` abstraction (Google Drive
+in Colab, local filesystem in tests). Any stage stops gracefully at the session
+time budget; re-running resumes exactly where it left off.
+
+## Highlights
+
+- **Resumable** — `recover()` heals stale state; each stage skips what's done.
+- **Idempotent** — safe to re-run as often as you like.
+- **Drive-backed** — SQLite DB, deduped PDFs, extracted text, exports, and a
+  live `status.json` heartbeat all live under `MyDrive/ky_court-WIP/`.
+- **Never auto-deletes on Drive** — original backups are retained.
+- **Polite + fault-tolerant** — paced API calls, exponential backoff, capped
+  retries, and a `DRY_RUN` mode for fully offline exercise.
+
+## Layout
+
+| Path | Purpose |
+|---|---|
+| `ky_court_pipeline.ipynb` | the deliverable (generated notebook) |
+| `notebook_builder/` | sources: `build.py` + one module per cell (`c00_config` … `c90_verify`) |
+| `tests/` | offline pytest suite (18 tests; live ones gated behind `KY_LIVE=1`) |
+| `scripts/smoke_local.py` | full offline pipeline over the real DB |
+| `RUNBOOK.md` | full operational runbook (first run, resuming, failure modes, config knobs) |
+
+## Quick start
+
+```bash
+# local test/smoke
+python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
+.venv/bin/pytest tests -q
+.venv/bin/python scripts/smoke_local.py
+.venv/bin/python notebook_builder/build.py   # regenerate the notebook
+
+# on Google Colab
+# upload ky_court-WIP-resume-*.tar.gz to MyDrive/ (if Drive is empty),
+# open ky_court_pipeline.ipynb, Runtime > Run all.
+```
+
+See **`RUNBOOK.md`** for the full runbook — first run, resuming, what "done"
+looks like, failure modes, and every config knob.
+
+> **Note:** input data (extracted archives, resume tarballs, `_WIP` backups)
+> is deliberately **not** in this repo — it lives in Google Drive / locally.