vault backup: 2026-08-12 15:38:55
This commit is contained in:
shit-vault
@@ -0,0 +1,207 @@
|
||||
---
|
||||
title: "kycourt-colab — Kentucky Appellate Court Corpus Pipeline (project notes)"
|
||||
status: "active"
|
||||
folder: "repo root (kycourt-colab)"
|
||||
tags: [kycourt, pipeline, colab, project-notes]
|
||||
created: "2026-08-12"
|
||||
updated: "2026-08-12"
|
||||
version: "1.1.0"
|
||||
---
|
||||
|
||||
# kycourt-colab — Kentucky Appellate Court Corpus Pipeline
|
||||
|
||||
## Overview
|
||||
|
||||
A single **generated Google Colab notebook** (`ky_court_pipeline.ipynb`) that
|
||||
restores, discovers, downloads, extracts, and verifies the full Kentucky
|
||||
appellate court document corpus (~**98.7k documents**). It is built to survive
|
||||
Colab's ephemeral runtimes: it auto-resumes after disconnects, is idempotent,
|
||||
and mirrors all state to Google Drive so nothing is lost between sessions.
|
||||
|
||||
**Remote:** https://github.com/stateofshit/kycourt-colab (private)
|
||||
**Local worktree:** `/home/user/00-incoming/kycourt-colab`
|
||||
|
||||
## What problem it solves
|
||||
|
||||
A batch archival crawl of public court documents over a ~10-hour-per-session
|
||||
horizon, on infrastructure (Colab) that kills your process mid-run. Requirements
|
||||
that fall out of that:
|
||||
|
||||
- Every session must pick up exactly where the last one stopped.
|
||||
- Re-running must be safe (idempotent) — no double-downloads, no data loss.
|
||||
- State must live somewhere durable (Drive), because the runtime is throwaway.
|
||||
- The crawl must be polite to a public API (paced, backoff, retries) and
|
||||
deduplicate aggressively (content-addressed storage).
|
||||
|
||||
## Architecture
|
||||
|
||||
The notebook is **not hand-written** — its cells are inlined from real Python
|
||||
modules in `notebook_builder/cells/` (one module per cell, `c00` → `c90`), and
|
||||
regenerated with `notebook_builder/build.py`. This means the pipeline is
|
||||
version-controlled, unit-testable Python that *just happens to be shipped as* a
|
||||
Colab notebook.
|
||||
|
||||
Two shared abstractions carry every stage:
|
||||
|
||||
- **SQLite DB** (`ky_court.sqlite`) — source of truth: `documents`,
|
||||
`discovery_runs`, `extraction_log`, `kv`. Opened in **WAL mode** with a
|
||||
30s `busy_timeout` so a background reader can snapshot it while the writer
|
||||
commits (see concurrency below).
|
||||
- **`Store` abstraction** — Google Drive in Colab, local filesystem in tests;
|
||||
exposes `exists / read_file / write_bytes / upload_file / list_dir`. All
|
||||
stages talk to it, never to Drive directly.
|
||||
|
||||
**Stage order:** `restore → discover → backfill → download → extract → sync/verify`.
|
||||
|
||||
## The six stages (what each actually does)
|
||||
|
||||
### 1. Restore — `c40_restore.py`
|
||||
Rehydrate a fresh/empty workspace from the newest `ky_court-WIP-resume-*.tar.gz`.
|
||||
Only acts when no DB exists; skips otherwise. If a stale workspace dir is
|
||||
present it is **moved aside to `-pre-restore` (renamed, never deleted)** before
|
||||
extracting — the "never destroys data" guarantee. Extracts to temp, then moves
|
||||
into place.
|
||||
|
||||
### 2. Discover — `c50_discover.py`
|
||||
Enumerate the corpus by searching the court API so we know *what exists*.
|
||||
Builds search shards (`YYYY-CA`, `YYYY-SC` per year; ceiling-hit shards get
|
||||
sub-sharded `YYYY-CA-0..9`). Pages results with `X-CTrack-Paging-*` headers up
|
||||
to a 10k ceiling; a `400` or ceiling-cross raises to force sub-sharding rather
|
||||
than silently missing docs. Normalizes rows into `documents` with
|
||||
`status='discovered'`, inserted idempotently via `ON CONFLICT(document_id)`.
|
||||
Sweeps repeatedly until a pass adds fewer than `CONVERGENCE_DELTA` docs
|
||||
(convergence). Every sweep logged to `discovery_runs`.
|
||||
|
||||
### 3. Backfill — `c60_backfill.py`
|
||||
Deduplicate *existing* backups into the content-addressed store. Hashes every
|
||||
file in `pdf_backup/` (SHA-256), writes unique content to `pdf_by_hash/<sha>.pdf`
|
||||
(9,582 backup files → 3,326 unique PDFs) — "do we already have this?" becomes a
|
||||
hash lookup. Normalizes DB `local_pdf` pointers. Guarded by `kv backfill_done`.
|
||||
|
||||
### 4. Download — `c70_download.py`
|
||||
Fetch PDFs for all `discovered`/`failed` docs with `attempts < MAX_ATTEMPTS`,
|
||||
in batches of 50. `api_download` does a `HEAD` (rejects non-200 and anything
|
||||
> `MAX_PDF_BYTES`), then streams `GET`, hashing as it goes. Dedup wins: a hash
|
||||
already in the store/manifest uploads nothing. **Retry/backoff** on transient
|
||||
errors: `sleep = min(60, 5·2^attempts)` (10→20→40→60s, capped), up to 4
|
||||
attempts. Permanent errors (400/401/403/404/**413**) fail immediately, no
|
||||
backoff. Writes to `tmp/<id>.part`, promotes to store, paces itself, and pushes
|
||||
a heartbeat to `status.json` every 100 attempts.
|
||||
|
||||
### 5. Extract — `c80_extract.py`
|
||||
Turn downloaded PDFs into searchable text with `pypdf`. Batches of 500 with 8
|
||||
parallel worker threads. A PDF with no text layer is a valid `"empty"` result,
|
||||
not an error. Text written to `text/<sha>.txt`; `extracted_chars` recorded.
|
||||
Heuristic flags likely-scanned (image-only) PDFs when `chars/pages < 20`.
|
||||
|
||||
### 6. Sync/Verify — `c90_verify.py`
|
||||
Prove nothing is corrupted and push an authoritative snapshot + summaries to
|
||||
Drive. Runs `PRAGMA integrity_check`; counts totals and the **remaining
|
||||
queues** (download + extract eligibility queries mirror the stage code exactly,
|
||||
so `done` is trustworthy). Exports `document_inventory.csv` +
|
||||
`failed_inventory.csv`. `checkpoint()` makes a consistent DB snapshot via
|
||||
`src.backup(dst)` (SQLite online backup API) and uploads it as the new
|
||||
`ky_court.sqlite`. Updates `status.json` with `done = (queues empty)`.
|
||||
Prints `PASS`/`FAIL`.
|
||||
|
||||
## Concurrency (the interesting part)
|
||||
|
||||
`checkpoint()` uses SQLite's **online backup API** (`src.backup(dst)`) from a
|
||||
**separate connection** — not `copytree`, not a WAL checkpoint. It copies the
|
||||
DB page-by-page and re-copies any page the writer modified during the pass,
|
||||
looping until a clean pass completes. Combined with **WAL mode** (readers don't
|
||||
block the writer) and **`busy_timeout=30000`**, this yields a
|
||||
transactionally-consistent snapshot even while Download is committing every doc.
|
||||
A naive file copy would produce a torn/corrupt snapshot and drop WAL-committed
|
||||
data.
|
||||
|
||||
## Key design decisions
|
||||
|
||||
- **Idempotent + resume-safe everywhere**: skip what's done, never auto-delete
|
||||
on Drive.
|
||||
- **Named permanent-failure set** (`PERMANENT_STATUSES` + `_is_permanent()`)
|
||||
distinguishes never-succeed (401/403/404/413) from transient (5xx/conn/429/408).
|
||||
- **Content-addressed store** (`pdf_by_hash/<sha>.pdf`) makes dedup a lookup.
|
||||
- **Config via one `CFG` dict** with env overrides (`KY_BUDGET`, `KY_DRY`,
|
||||
`KY_PHASES`, `KY_RETRY_HARD`, `KY_LIVE`).
|
||||
|
||||
## Config knobs (top cell `CFG`)
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `BUDGET_SECONDS` | 10h | max wall-clock per session (`KY_BUDGET`) |
|
||||
| `SHUTDOWN_SECONDS` | 15 min | stop early so the session finishes cleanly |
|
||||
| `PACING_SEARCH` / `PACING_DL` | (0.3,0.7)/(0.3,0.5) | random sleep between API calls |
|
||||
| `MAX_ATTEMPTS` | 4 | download retries before `failed` |
|
||||
| `MAX_PDF_BYTES` | 200 MiB | size cap; oversized (413) fails permanently, no retry/backoff |
|
||||
| `MAX_EXTRACT_ATTEMPTS` | 3 | extraction retries before skip |
|
||||
| `RETRY_HARD` | 0 (`KY_RETRY_HARD`) | on resume, reset `failed` → `discovered` |
|
||||
| `PHASES` | discover,backfill,download,extract | stages to run (`KY_PHASES`) |
|
||||
| `DRY_RUN` | 0 (`KY_DRY`) | 1 = fully offline, no network |
|
||||
|
||||
## File layout
|
||||
|
||||
```
|
||||
ky_court_pipeline.ipynb the deliverable (generated)
|
||||
notebook_builder/
|
||||
build.py regenerate the notebook from cells
|
||||
cells/ c00_config … c90_verify + drive_store.py
|
||||
tests/ test_pipeline_local.py + conftest.py (offline; live gated by KY_LIVE)
|
||||
scripts/ smoke_local.py full offline pipeline over the real DB
|
||||
RUNBOOK.md operational runbook (first run, resuming, failure modes)
|
||||
docs/superpowers/ specs/ + plans/ design spec + implementation plan
|
||||
PROJECT_NOTES.md this file
|
||||
.gitignore ignores .venv, extracted/, *.sqlite, __pycache__,
|
||||
.worktrees/, .superpowers/ (NOT the WIP tars)
|
||||
```
|
||||
|
||||
> **Data is NOT in the repo.** `extracted/`, `*.tar.gz` resume/WIP archives, and
|
||||
> `sanitize-work/` live only in Drive / locally. Gitignore does not cover the
|
||||
> WIP `.tar.gz` files, so never `git add -A` from the repo root.
|
||||
|
||||
## Running / testing
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
|
||||
.venv/bin/pytest tests -q # 20 offline tests (live ones skipped)
|
||||
.venv/bin/python scripts/smoke_local.py # full pipeline over the real 98,724-row DB
|
||||
.venv/bin/python notebook_builder/build.py # regenerate the notebook
|
||||
```
|
||||
|
||||
Live API tests are gated behind `KY_LIVE=1` and share a ≤10-request budget.
|
||||
Suite result at v1.2: **20 passed, 3 skipped**.
|
||||
|
||||
## Failure modes (from RUNBOOK)
|
||||
|
||||
- **Colab disconnect mid-run** → re-run; `recover()` heals stale rows, drops
|
||||
`.part` files.
|
||||
- **Drive quota exceeded** → checkpoint push fails; run continues, store lags.
|
||||
- **5xx/connection storms** → backoff `min(60, 5·2^attempts)` up to `MAX_ATTEMPTS`.
|
||||
- **Search shard hits 10k ceiling** → planner sub-shards next run.
|
||||
- **`integrity_check` not "ok"** → DB corrupt; restore from last resume tar.
|
||||
|
||||
## Versioning / releases
|
||||
|
||||
Immutable milestone tags (never force-move a published tag):
|
||||
|
||||
```
|
||||
pipeline-v1 33a6ec1 original pipeline milestone
|
||||
v1.1 ae77344 +README
|
||||
v1.2 ec248fe +413 oversize permanent-failure fix (MAX_PDF_BYTES knob)
|
||||
```
|
||||
|
||||
`v1.2` also has a **GitHub release** with changelog:
|
||||
https://github.com/stateofshit/kycourt-colab/releases/tag/v1.2
|
||||
|
||||
## Status
|
||||
|
||||
Project complete and shipped. Suite green offline. The pipeline is ready to run in Colab (restore from the resume tar → Run all).
|
||||
|
||||
### Roadmap / Future enhancements
|
||||
- **Parallel extract workers** — tune the 8-worker batch size based on observed Colab runtime limits.
|
||||
- **Drive quota monitoring** — automated alert when `status.json` reports approaching quota, with graceful degradation to skip upload.
|
||||
- **CLI entry point** — `kycourt-pipeline run --phase discover` to run individual stages outside Colab.
|
||||
- **Query layer** — lightweight read-only API over the SQLite DB for inspecting discovered/extracted docs without re-running the pipeline.
|
||||
- **Vault migration** — if/when the repo is folded into the vault, adapt `folder` frontmatter and consolidate `.tar.gz` WIP archives under the vault's resume format. Remote backup, README, tags, and v1.2 release all
|
||||
in place on GitHub. Suite green offline. The pipeline is ready to run in Colab
|
||||
(restore from the resume tar → Run all).
|
||||
Reference in New Issue
Block a user