117 lines
5.7 KiB
Markdown
117 lines
5.7 KiB
Markdown
# RUNBOOK — ky_court_pipeline.ipynb
|
||
|
||
One notebook to restore, discover, download, extract, sync, and verify the full
|
||
Kentucky appellate court document corpus (~98.7k documents). It is designed to
|
||
run in Google Colab, auto-resume after disconnects, and mirror all state up to a
|
||
Drive folder so nothing is lost across sessions.
|
||
|
||
## What this is
|
||
|
||
A single generated notebook (`ky_court_pipeline.ipynb`) whose cells are inlined
|
||
from real Python modules in `notebook_builder/cells/`. The notebook is rebuilt
|
||
with `notebook_builder/build.py`. All stages read/write the same SQLite DB and a
|
||
`Store` abstraction that is Google Drive in Colab and the local filesystem in
|
||
tests/smoke.
|
||
|
||
Stage order: **restore → discover → backfill → download → extract → sync/verify**.
|
||
Any stage stops gracefully at the session budget; re-running resumes where it
|
||
left off.
|
||
|
||
## Files on Drive (`MyDrive/ky_court-WIP/`)
|
||
|
||
| Path | Purpose |
|
||
|---|---|
|
||
| `ky_court.sqlite` | source of truth (documents, discovery_runs, kv, extraction_log) |
|
||
| `pdf_by_hash/<sha256>.pdf` | deduped PDFs keyed by content hash (3,326 unique from 9,582 backup files) |
|
||
| `pdf_backup/<case>_<n>.pdf` | original named backups — **retained, never deleted** |
|
||
| `text/<sha256>.txt` | extracted text per PDF |
|
||
| `exports/file_manifest.csv` | sha256 → size manifest |
|
||
| `exports/document_inventory.csv` / `failed_inventory.csv` | exported summaries from verify |
|
||
| `status.json` | live heartbeat + phase/counts (the "is it alive" check) |
|
||
| `ky_court-WIP-resume-*.tar.gz` | (MyDrive root) resume tarball used to rehydrate a fresh runtime |
|
||
|
||
The store root is `MyDrive/ky_court-WIP` (Colab) or `<WORKDIR>/ky_court-WIP` (local).
|
||
|
||
## First run
|
||
|
||
1. If Drive is empty (no `ky_court-WIP`), upload `ky_court-WIP-resume-YYYY-MM-DD.tar.gz`
|
||
to `MyDrive/`. Stage **restore** will extract it.
|
||
2. Open `ky_court_pipeline.ipynb` in Colab.
|
||
3. Edit the first code cell (`CFG`) if needed (budget, phases — see Config knobs).
|
||
4. **Runtime > Run all**. A session mounts Drive, opens the DB (pulling it from
|
||
the store if absent), heals stale state, then runs the enabled phases.
|
||
|
||
Expected session timeline (typical):
|
||
- setup: ~1 min (Drive mount, dirs, recover)
|
||
- discover: 10–30 min (74 shards, paged search, paced)
|
||
- backfill: ~20 min (hash `pdf_backup` → dedupe into `pdf_by_hash`)
|
||
- download: ~10 h per session ≈ 6–12k documents (budget-capped)
|
||
- extract: rides along between batches (thread-pooled PDF text extraction)
|
||
- verify: ~5 min (integrity check, inventory exports, status push)
|
||
|
||
## Resuming
|
||
|
||
Just re-run the notebook. `recover()` closes any stale `running` discovery runs,
|
||
drops leftover `.part` files, and each stage skips what's already done:
|
||
- backfill: guarded by `kv backfill_done`
|
||
- download: only `discovered`/`failed` docs with `attempts < MAX_ATTEMPTS`
|
||
- extract: only `downloaded` docs with no extracted text yet
|
||
|
||
Idempotent by design — safe to rerun as often as you like.
|
||
|
||
## What "done" looks like
|
||
|
||
Verify prints `PASS {checks}` and, when every queue is empty:
|
||
- notebook verify output includes `done: True`
|
||
- `status.json` on Drive contains `"done": true`
|
||
|
||
A session that merely hit its budget is **not** "done" — it stops early with
|
||
`status.json` showing the phase and remaining queues. That is normal mid-flight
|
||
state, not an error.
|
||
|
||
## Failure modes
|
||
|
||
- **Colab disconnect mid-run** → just re-run. `recover()` heals stale rows;
|
||
`.part` files are dropped; partial downloads retry from the queue.
|
||
- **Drive quota exceeded** → verify/checkpoint push fails; the run continues
|
||
locally in the session but the store lags. Check Drive quota, free space, re-run.
|
||
- **5xx / connection storms** → `api_download` returns an error tuple and
|
||
`stage_download` backs off `min(60, 5*2^attempts)` seconds, retrying up to
|
||
`MAX_ATTEMPTS` before marking the doc `failed`.
|
||
- **Search shard hits the 10k ceiling** → `api_search_pages` raises; the planner
|
||
sub-shards that seed (`YYYY-CA-N`) on the next run.
|
||
- **`integrity` check not "ok"** → verify prints `FAIL`; the DB is corrupted and
|
||
should be restored from the last resume tar before trusting anything.
|
||
|
||
## Config knobs (top code cell `CFG`)
|
||
|
||
| Key | Default | Meaning |
|
||
|---|---|---|
|
||
| `BUDGET_SECONDS` | 10 * 3600 | max wall-clock per session (env `KY_BUDGET`) |
|
||
| `SHUTDOWN_SECONDS` | 900 | stop stages this long before the deadline so the session can finish cleanly |
|
||
| `PACING_SEARCH` / `PACING_DL` | (0.3,0.7) / (0.3,0.5) | random sleep range between API calls (polite throttling) |
|
||
| `MAX_ATTEMPTS` | 4 | download retries before `failed` |
|
||
| `MAX_PDF_BYTES` | 200 MiB | size cap; oversized (HTTP 413) fails permanently, no retry/backoff |
|
||
| `MAX_EXTRACT_ATTEMPTS` | 3 | PDF-extraction retries before skip |
|
||
| `RETRY_HARD` | 0 (env `KY_RETRY_HARD`) | on resume, reset `failed` docs back to `discovered` |
|
||
| `PHASES` | discover,backfill,download,extract | comma-separated phases to run (env `KY_PHASES`) |
|
||
| `DRY_RUN` | 0 (env `KY_DRY`) | 1 = fully offline: discovery/download skip all network |
|
||
|
||
## Safety notes
|
||
|
||
- The pipeline **never deletes on Drive automatically**. `pdf_backup` is retained;
|
||
pre-restore move-aside only renames (`-pre-restore`), never deletes.
|
||
- Set `DRY_RUN=1` (`KY_DRY=1`) to exercise the whole flow offline with zero
|
||
network calls — the smoke script (`scripts/smoke_local.py`) does exactly this.
|
||
- Live API tests are gated behind `KY_LIVE=1` and share a ≤10-request budget;
|
||
keep them there so the ordinary test suite stays offline and fast.
|
||
|
||
## Local / tests
|
||
|
||
```bash
|
||
python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
|
||
.venv/bin/pytest tests -q # 18 offline tests (live ones skipped)
|
||
.venv/bin/python scripts/smoke_local.py # full pipeline over the real 98,724-row DB
|
||
.venv/bin/python notebook_builder/build.py # regenerate the notebook
|
||
```
|