hollywood/zz-MAIN/README.md

ky-court-colab

A single generated Google Colab notebook that restores, discovers, downloads, extracts, and verifies the full Kentucky appellate court document corpus (~98.7k documents). Built to survive Colab disconnects — it auto-resumes, is idempotent, and mirrors all state to Google Drive so nothing is lost between sessions.

What it does

ky_court_pipeline.ipynb walks the corpus through six stages:

restore → discover → backfill → download → extract → sync/verify

Every stage reads/writes one SQLite DB and a Store abstraction (Google Drive in Colab, local filesystem in tests). Any stage stops gracefully at the session time budget; re-running resumes exactly where it left off.

Highlights

• Resumable — recover() heals stale state; each stage skips what's done.
• Idempotent — safe to re-run as often as you like.
• Drive-backed — SQLite DB, deduped PDFs, extracted text, exports, and a live status.json heartbeat all live under MyDrive/ky_court-WIP/.
• Never auto-deletes on Drive — original backups are retained.
• Polite + fault-tolerant — paced API calls, exponential backoff, capped retries, and a DRY_RUN mode for fully offline exercise.

Layout

Path	Purpose
ky_court_pipeline.ipynb	the deliverable (generated notebook)
notebook_builder/	sources: build.py + one module per cell (c00_config … c90_verify)
tests/	offline pytest suite (18 tests; live ones gated behind KY_LIVE=1)
scripts/smoke_local.py	full offline pipeline over the real DB
RUNBOOK.md	full operational runbook (first run, resuming, failure modes, config knobs)

Quick start

# local test/smoke
python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
.venv/bin/pytest tests -q
.venv/bin/python scripts/smoke_local.py
.venv/bin/python notebook_builder/build.py   # regenerate the notebook

# on Google Colab
# upload ky_court-WIP-resume-*.tar.gz to MyDrive/ (if Drive is empty),
# open ky_court_pipeline.ipynb, Runtime > Run all.

See RUNBOOK.md for the full runbook — first run, resuming, what "done" looks like, failure modes, and every config knob.

Note: input data (extracted archives, resume tarballs, _WIP backups) is deliberately not in this repo — it lives in Google Drive / locally.