# RUNBOOK — ky_court_pipeline.ipynb

One notebook to restore, discover, download, extract, sync, and verify the full
Kentucky appellate court document corpus (~98.7k documents). It is designed to
run in Google Colab, auto-resume after disconnects, and mirror all state up to a
Drive folder so nothing is lost across sessions.

## What this is

A single generated notebook (`ky_court_pipeline.ipynb`) whose cells are inlined
from real Python modules in `notebook_builder/cells/`. The notebook is rebuilt
with `notebook_builder/build.py`. All stages read/write the same SQLite DB and a
`Store` abstraction that is Google Drive in Colab and the local filesystem in
tests/smoke.

Stage order: **restore → discover → backfill → download → extract → sync/verify**.
Any stage stops gracefully at the session budget; re-running resumes where it
left off.

## Files on Drive (`MyDrive/ky_court-WIP/`)

| Path | Purpose |
|---|---|
| `ky_court.sqlite` | source of truth (documents, discovery_runs, kv, extraction_log) |
| `pdf_by_hash/<sha256>.pdf` | deduped PDFs keyed by content hash (3,326 unique from 9,582 backup files) |
| `pdf_backup/<case>_<n>.pdf` | original named backups — **retained, never deleted** |
| `text/<sha256>.txt` | extracted text per PDF |
| `exports/file_manifest.csv` | sha256 → size manifest |
| `exports/document_inventory.csv` / `failed_inventory.csv` | exported summaries from verify |
| `status.json` | live heartbeat + phase/counts (the "is it alive" check) |
| `ky_court-WIP-resume-*.tar.gz` | (MyDrive root) resume tarball used to rehydrate a fresh runtime |

The store root is `MyDrive/ky_court-WIP` (Colab) or `<WORKDIR>/ky_court-WIP` (local).

## First run

1. If Drive is empty (no `ky_court-WIP`), upload `ky_court-WIP-resume-YYYY-MM-DD.tar.gz`
   to `MyDrive/`. Stage **restore** will extract it.
2. Open `ky_court_pipeline.ipynb` in Colab.
3. Edit the first code cell (`CFG`) if needed (budget, phases — see Config knobs).
4. **Runtime > Run all**. A session mounts Drive, opens the DB (pulling it from
   the store if absent), heals stale state, then runs the enabled phases.

Expected session timeline (typical):
- setup: ~1 min (Drive mount, dirs, recover)
- discover: 10–30 min (74 shards, paged search, paced)
- backfill: ~20 min (hash `pdf_backup` → dedupe into `pdf_by_hash`)
- download: ~10 h per session ≈ 6–12k documents (budget-capped)
- extract: rides along between batches (thread-pooled PDF text extraction)
- verify: ~5 min (integrity check, inventory exports, status push)

## Resuming

Just re-run the notebook. `recover()` closes any stale `running` discovery runs,
drops leftover `.part` files, and each stage skips what's already done:
- backfill: guarded by `kv backfill_done`
- download: only `discovered`/`failed` docs with `attempts < MAX_ATTEMPTS`
- extract: only `downloaded` docs with no extracted text yet

Idempotent by design — safe to rerun as often as you like.

## What "done" looks like

Verify prints `PASS {checks}` and, when every queue is empty:
- notebook verify output includes `done: True`
- `status.json` on Drive contains `"done": true`

A session that merely hit its budget is **not** "done" — it stops early with
`status.json` showing the phase and remaining queues. That is normal mid-flight
state, not an error.

## Failure modes

- **Colab disconnect mid-run** → just re-run. `recover()` heals stale rows;
  `.part` files are dropped; partial downloads retry from the queue.
- **Drive quota exceeded** → verify/checkpoint push fails; the run continues
  locally in the session but the store lags. Check Drive quota, free space, re-run.
- **5xx / connection storms** → `api_download` returns an error tuple and
  `stage_download` backs off `min(60, 5*2^attempts)` seconds, retrying up to
  `MAX_ATTEMPTS` before marking the doc `failed`.
- **Search shard hits the 10k ceiling** → `api_search_pages` raises; the planner
  sub-shards that seed (`YYYY-CA-N`) on the next run.
- **`integrity` check not "ok"** → verify prints `FAIL`; the DB is corrupted and
  should be restored from the last resume tar before trusting anything.

## Config knobs (top code cell `CFG`)

| Key | Default | Meaning |
|---|---|---|
| `BUDGET_SECONDS` | 10 * 3600 | max wall-clock per session (env `KY_BUDGET`) |
| `SHUTDOWN_SECONDS` | 900 | stop stages this long before the deadline so the session can finish cleanly |
| `PACING_SEARCH` / `PACING_DL` | (0.3,0.7) / (0.3,0.5) | random sleep range between API calls (polite throttling) |
| `MAX_ATTEMPTS` | 4 | download retries before `failed` |
| `MAX_PDF_BYTES` | 200 MiB | size cap; oversized (HTTP 413) fails permanently, no retry/backoff |
| `MAX_EXTRACT_ATTEMPTS` | 3 | PDF-extraction retries before skip |
| `RETRY_HARD` | 0 (env `KY_RETRY_HARD`) | on resume, reset `failed` docs back to `discovered` |
| `PHASES` | discover,backfill,download,extract | comma-separated phases to run (env `KY_PHASES`) |
| `DRY_RUN` | 0 (env `KY_DRY`) | 1 = fully offline: discovery/download skip all network |

## Safety notes

- The pipeline **never deletes on Drive automatically**. `pdf_backup` is retained;
  pre-restore move-aside only renames (`-pre-restore`), never deletes.
- Set `DRY_RUN=1` (`KY_DRY=1`) to exercise the whole flow offline with zero
  network calls — the smoke script (`scripts/smoke_local.py`) does exactly this.
- Live API tests are gated behind `KY_LIVE=1` and share a ≤10-request budget;
  keep them there so the ordinary test suite stays offline and fast.

## Local / tests

```bash
python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
.venv/bin/pytest tests -q               # 18 offline tests (live ones skipped)
.venv/bin/python scripts/smoke_local.py # full pipeline over the real 98,724-row DB
.venv/bin/python notebook_builder/build.py  # regenerate the notebook
```