208 lines
10 KiB
Markdown
208 lines
10 KiB
Markdown
---
|
|
title: "kycourt-colab — Kentucky Appellate Court Corpus Pipeline (project notes)"
|
|
status: "active"
|
|
folder: "repo root (kycourt-colab)"
|
|
tags: [kycourt, pipeline, colab, project-notes]
|
|
created: "2026-08-12"
|
|
updated: "2026-08-12"
|
|
version: "1.1.0"
|
|
---
|
|
|
|
# kycourt-colab — Kentucky Appellate Court Corpus Pipeline
|
|
|
|
## Overview
|
|
|
|
A single **generated Google Colab notebook** (`ky_court_pipeline.ipynb`) that
|
|
restores, discovers, downloads, extracts, and verifies the full Kentucky
|
|
appellate court document corpus (~**98.7k documents**). It is built to survive
|
|
Colab's ephemeral runtimes: it auto-resumes after disconnects, is idempotent,
|
|
and mirrors all state to Google Drive so nothing is lost between sessions.
|
|
|
|
**Remote:** https://github.com/stateofshit/kycourt-colab (private)
|
|
**Local worktree:** `/home/user/00-incoming/kycourt-colab`
|
|
|
|
## What problem it solves
|
|
|
|
A batch archival crawl of public court documents over a ~10-hour-per-session
|
|
horizon, on infrastructure (Colab) that kills your process mid-run. Requirements
|
|
that fall out of that:
|
|
|
|
- Every session must pick up exactly where the last one stopped.
|
|
- Re-running must be safe (idempotent) — no double-downloads, no data loss.
|
|
- State must live somewhere durable (Drive), because the runtime is throwaway.
|
|
- The crawl must be polite to a public API (paced, backoff, retries) and
|
|
deduplicate aggressively (content-addressed storage).
|
|
|
|
## Architecture
|
|
|
|
The notebook is **not hand-written** — its cells are inlined from real Python
|
|
modules in `notebook_builder/cells/` (one module per cell, `c00` → `c90`), and
|
|
regenerated with `notebook_builder/build.py`. This means the pipeline is
|
|
version-controlled, unit-testable Python that *just happens to be shipped as* a
|
|
Colab notebook.
|
|
|
|
Two shared abstractions carry every stage:
|
|
|
|
- **SQLite DB** (`ky_court.sqlite`) — source of truth: `documents`,
|
|
`discovery_runs`, `extraction_log`, `kv`. Opened in **WAL mode** with a
|
|
30s `busy_timeout` so a background reader can snapshot it while the writer
|
|
commits (see concurrency below).
|
|
- **`Store` abstraction** — Google Drive in Colab, local filesystem in tests;
|
|
exposes `exists / read_file / write_bytes / upload_file / list_dir`. All
|
|
stages talk to it, never to Drive directly.
|
|
|
|
**Stage order:** `restore → discover → backfill → download → extract → sync/verify`.
|
|
|
|
## The six stages (what each actually does)
|
|
|
|
### 1. Restore — `c40_restore.py`
|
|
Rehydrate a fresh/empty workspace from the newest `ky_court-WIP-resume-*.tar.gz`.
|
|
Only acts when no DB exists; skips otherwise. If a stale workspace dir is
|
|
present it is **moved aside to `-pre-restore` (renamed, never deleted)** before
|
|
extracting — the "never destroys data" guarantee. Extracts to temp, then moves
|
|
into place.
|
|
|
|
### 2. Discover — `c50_discover.py`
|
|
Enumerate the corpus by searching the court API so we know *what exists*.
|
|
Builds search shards (`YYYY-CA`, `YYYY-SC` per year; ceiling-hit shards get
|
|
sub-sharded `YYYY-CA-0..9`). Pages results with `X-CTrack-Paging-*` headers up
|
|
to a 10k ceiling; a `400` or ceiling-cross raises to force sub-sharding rather
|
|
than silently missing docs. Normalizes rows into `documents` with
|
|
`status='discovered'`, inserted idempotently via `ON CONFLICT(document_id)`.
|
|
Sweeps repeatedly until a pass adds fewer than `CONVERGENCE_DELTA` docs
|
|
(convergence). Every sweep logged to `discovery_runs`.
|
|
|
|
### 3. Backfill — `c60_backfill.py`
|
|
Deduplicate *existing* backups into the content-addressed store. Hashes every
|
|
file in `pdf_backup/` (SHA-256), writes unique content to `pdf_by_hash/<sha>.pdf`
|
|
(9,582 backup files → 3,326 unique PDFs) — "do we already have this?" becomes a
|
|
hash lookup. Normalizes DB `local_pdf` pointers. Guarded by `kv backfill_done`.
|
|
|
|
### 4. Download — `c70_download.py`
|
|
Fetch PDFs for all `discovered`/`failed` docs with `attempts < MAX_ATTEMPTS`,
|
|
in batches of 50. `api_download` does a `HEAD` (rejects non-200 and anything
|
|
> `MAX_PDF_BYTES`), then streams `GET`, hashing as it goes. Dedup wins: a hash
|
|
already in the store/manifest uploads nothing. **Retry/backoff** on transient
|
|
errors: `sleep = min(60, 5·2^attempts)` (10→20→40→60s, capped), up to 4
|
|
attempts. Permanent errors (400/401/403/404/**413**) fail immediately, no
|
|
backoff. Writes to `tmp/<id>.part`, promotes to store, paces itself, and pushes
|
|
a heartbeat to `status.json` every 100 attempts.
|
|
|
|
### 5. Extract — `c80_extract.py`
|
|
Turn downloaded PDFs into searchable text with `pypdf`. Batches of 500 with 8
|
|
parallel worker threads. A PDF with no text layer is a valid `"empty"` result,
|
|
not an error. Text written to `text/<sha>.txt`; `extracted_chars` recorded.
|
|
Heuristic flags likely-scanned (image-only) PDFs when `chars/pages < 20`.
|
|
|
|
### 6. Sync/Verify — `c90_verify.py`
|
|
Prove nothing is corrupted and push an authoritative snapshot + summaries to
|
|
Drive. Runs `PRAGMA integrity_check`; counts totals and the **remaining
|
|
queues** (download + extract eligibility queries mirror the stage code exactly,
|
|
so `done` is trustworthy). Exports `document_inventory.csv` +
|
|
`failed_inventory.csv`. `checkpoint()` makes a consistent DB snapshot via
|
|
`src.backup(dst)` (SQLite online backup API) and uploads it as the new
|
|
`ky_court.sqlite`. Updates `status.json` with `done = (queues empty)`.
|
|
Prints `PASS`/`FAIL`.
|
|
|
|
## Concurrency (the interesting part)
|
|
|
|
`checkpoint()` uses SQLite's **online backup API** (`src.backup(dst)`) from a
|
|
**separate connection** — not `copytree`, not a WAL checkpoint. It copies the
|
|
DB page-by-page and re-copies any page the writer modified during the pass,
|
|
looping until a clean pass completes. Combined with **WAL mode** (readers don't
|
|
block the writer) and **`busy_timeout=30000`**, this yields a
|
|
transactionally-consistent snapshot even while Download is committing every doc.
|
|
A naive file copy would produce a torn/corrupt snapshot and drop WAL-committed
|
|
data.
|
|
|
|
## Key design decisions
|
|
|
|
- **Idempotent + resume-safe everywhere**: skip what's done, never auto-delete
|
|
on Drive.
|
|
- **Named permanent-failure set** (`PERMANENT_STATUSES` + `_is_permanent()`)
|
|
distinguishes never-succeed (401/403/404/413) from transient (5xx/conn/429/408).
|
|
- **Content-addressed store** (`pdf_by_hash/<sha>.pdf`) makes dedup a lookup.
|
|
- **Config via one `CFG` dict** with env overrides (`KY_BUDGET`, `KY_DRY`,
|
|
`KY_PHASES`, `KY_RETRY_HARD`, `KY_LIVE`).
|
|
|
|
## Config knobs (top cell `CFG`)
|
|
|
|
| Key | Default | Meaning |
|
|
|---|---|---|
|
|
| `BUDGET_SECONDS` | 10h | max wall-clock per session (`KY_BUDGET`) |
|
|
| `SHUTDOWN_SECONDS` | 15 min | stop early so the session finishes cleanly |
|
|
| `PACING_SEARCH` / `PACING_DL` | (0.3,0.7)/(0.3,0.5) | random sleep between API calls |
|
|
| `MAX_ATTEMPTS` | 4 | download retries before `failed` |
|
|
| `MAX_PDF_BYTES` | 200 MiB | size cap; oversized (413) fails permanently, no retry/backoff |
|
|
| `MAX_EXTRACT_ATTEMPTS` | 3 | extraction retries before skip |
|
|
| `RETRY_HARD` | 0 (`KY_RETRY_HARD`) | on resume, reset `failed` → `discovered` |
|
|
| `PHASES` | discover,backfill,download,extract | stages to run (`KY_PHASES`) |
|
|
| `DRY_RUN` | 0 (`KY_DRY`) | 1 = fully offline, no network |
|
|
|
|
## File layout
|
|
|
|
```
|
|
ky_court_pipeline.ipynb the deliverable (generated)
|
|
notebook_builder/
|
|
build.py regenerate the notebook from cells
|
|
cells/ c00_config … c90_verify + drive_store.py
|
|
tests/ test_pipeline_local.py + conftest.py (offline; live gated by KY_LIVE)
|
|
scripts/ smoke_local.py full offline pipeline over the real DB
|
|
RUNBOOK.md operational runbook (first run, resuming, failure modes)
|
|
docs/superpowers/ specs/ + plans/ design spec + implementation plan
|
|
PROJECT_NOTES.md this file
|
|
.gitignore ignores .venv, extracted/, *.sqlite, __pycache__,
|
|
.worktrees/, .superpowers/ (NOT the WIP tars)
|
|
```
|
|
|
|
> **Data is NOT in the repo.** `extracted/`, `*.tar.gz` resume/WIP archives, and
|
|
> `sanitize-work/` live only in Drive / locally. Gitignore does not cover the
|
|
> WIP `.tar.gz` files, so never `git add -A` from the repo root.
|
|
|
|
## Running / testing
|
|
|
|
```bash
|
|
python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
|
|
.venv/bin/pytest tests -q # 20 offline tests (live ones skipped)
|
|
.venv/bin/python scripts/smoke_local.py # full pipeline over the real 98,724-row DB
|
|
.venv/bin/python notebook_builder/build.py # regenerate the notebook
|
|
```
|
|
|
|
Live API tests are gated behind `KY_LIVE=1` and share a ≤10-request budget.
|
|
Suite result at v1.2: **20 passed, 3 skipped**.
|
|
|
|
## Failure modes (from RUNBOOK)
|
|
|
|
- **Colab disconnect mid-run** → re-run; `recover()` heals stale rows, drops
|
|
`.part` files.
|
|
- **Drive quota exceeded** → checkpoint push fails; run continues, store lags.
|
|
- **5xx/connection storms** → backoff `min(60, 5·2^attempts)` up to `MAX_ATTEMPTS`.
|
|
- **Search shard hits 10k ceiling** → planner sub-shards next run.
|
|
- **`integrity_check` not "ok"** → DB corrupt; restore from last resume tar.
|
|
|
|
## Versioning / releases
|
|
|
|
Immutable milestone tags (never force-move a published tag):
|
|
|
|
```
|
|
pipeline-v1 33a6ec1 original pipeline milestone
|
|
v1.1 ae77344 +README
|
|
v1.2 ec248fe +413 oversize permanent-failure fix (MAX_PDF_BYTES knob)
|
|
```
|
|
|
|
`v1.2` also has a **GitHub release** with changelog:
|
|
https://github.com/stateofshit/kycourt-colab/releases/tag/v1.2
|
|
|
|
## Status
|
|
|
|
Project complete and shipped. Suite green offline. The pipeline is ready to run in Colab (restore from the resume tar → Run all).
|
|
|
|
### Roadmap / Future enhancements
|
|
- **Parallel extract workers** — tune the 8-worker batch size based on observed Colab runtime limits.
|
|
- **Drive quota monitoring** — automated alert when `status.json` reports approaching quota, with graceful degradation to skip upload.
|
|
- **CLI entry point** — `kycourt-pipeline run --phase discover` to run individual stages outside Colab.
|
|
- **Query layer** — lightweight read-only API over the SQLite DB for inspecting discovered/extracted docs without re-running the pipeline.
|
|
- **Vault migration** — if/when the repo is folded into the vault, adapt `folder` frontmatter and consolidate `.tar.gz` WIP archives under the vault's resume format. Remote backup, README, tags, and v1.2 release all
|
|
in place on GitHub. Suite green offline. The pipeline is ready to run in Colab
|
|
(restore from the resume tar → Run all).
|