# ky-court-colab

A single generated Google Colab notebook that **restores, discovers, downloads,
extracts, and verifies** the full Kentucky appellate court document corpus
(~98.7k documents). Built to survive Colab disconnects — it auto-resumes, is
idempotent, and mirrors all state to Google Drive so nothing is lost between
sessions.

## What it does

`ky_court_pipeline.ipynb` walks the corpus through six stages:

```
restore → discover → backfill → download → extract → sync/verify
```

Every stage reads/writes one SQLite DB and a `Store` abstraction (Google Drive
in Colab, local filesystem in tests). Any stage stops gracefully at the session
time budget; re-running resumes exactly where it left off.

## Highlights

- **Resumable** — `recover()` heals stale state; each stage skips what's done.
- **Idempotent** — safe to re-run as often as you like.
- **Drive-backed** — SQLite DB, deduped PDFs, extracted text, exports, and a
  live `status.json` heartbeat all live under `MyDrive/ky_court-WIP/`.
- **Never auto-deletes on Drive** — original backups are retained.
- **Polite + fault-tolerant** — paced API calls, exponential backoff, capped
  retries, and a `DRY_RUN` mode for fully offline exercise.

## Layout

| Path | Purpose |
|---|---|
| `ky_court_pipeline.ipynb` | the deliverable (generated notebook) |
| `notebook_builder/` | sources: `build.py` + one module per cell (`c00_config` … `c90_verify`) |
| `tests/` | offline pytest suite (18 tests; live ones gated behind `KY_LIVE=1`) |
| `scripts/smoke_local.py` | full offline pipeline over the real DB |
| `RUNBOOK.md` | full operational runbook (first run, resuming, failure modes, config knobs) |

## Quick start

```bash
# local test/smoke
python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
.venv/bin/pytest tests -q
.venv/bin/python scripts/smoke_local.py
.venv/bin/python notebook_builder/build.py   # regenerate the notebook

# on Google Colab
# upload ky_court-WIP-resume-*.tar.gz to MyDrive/ (if Drive is empty),
# open ky_court_pipeline.ipynb, Runtime > Run all.
```

See **`RUNBOOK.md`** for the full runbook — first run, resuming, what "done"
looks like, failure modes, and every config knob.

> **Note:** input data (extracted archives, resume tarballs, `_WIP` backups)
> is deliberately **not** in this repo — it lives in Google Drive / locally.