hollywood/zz-MAIN/PROJECT_NOTES.md

title, status, folder, tags, created, updated, version

title	status	folder	tags	created	updated	version
kycourt-colab — Kentucky Appellate Court Corpus Pipeline (project notes)	active	repo root (kycourt-colab)	kycourt pipeline colab project-notes	2026-08-12	2026-08-12	1.1.0

kycourt-colab — Kentucky Appellate Court Corpus Pipeline

Overview

A single generated Google Colab notebook (ky_court_pipeline.ipynb) that restores, discovers, downloads, extracts, and verifies the full Kentucky appellate court document corpus (~98.7k documents). It is built to survive Colab's ephemeral runtimes: it auto-resumes after disconnects, is idempotent, and mirrors all state to Google Drive so nothing is lost between sessions.

Remote: https://github.com/stateofshit/kycourt-colab (private) Local worktree: /home/user/00-incoming/kycourt-colab

What problem it solves

A batch archival crawl of public court documents over a ~10-hour-per-session horizon, on infrastructure (Colab) that kills your process mid-run. Requirements that fall out of that:

• Every session must pick up exactly where the last one stopped.
• Re-running must be safe (idempotent) — no double-downloads, no data loss.
• State must live somewhere durable (Drive), because the runtime is throwaway.
• The crawl must be polite to a public API (paced, backoff, retries) and deduplicate aggressively (content-addressed storage).

Architecture

The notebook is not hand-written — its cells are inlined from real Python modules in notebook_builder/cells/ (one module per cell, c00 → c90), and regenerated with notebook_builder/build.py. This means the pipeline is version-controlled, unit-testable Python that just happens to be shipped as a Colab notebook.

Two shared abstractions carry every stage:

• SQLite DB (ky_court.sqlite) — source of truth: documents, discovery_runs, extraction_log, kv. Opened in WAL mode with a 30s busy_timeout so a background reader can snapshot it while the writer commits (see concurrency below).
• Store abstraction — Google Drive in Colab, local filesystem in tests; exposes exists / read_file / write_bytes / upload_file / list_dir. All stages talk to it, never to Drive directly.

Stage order: restore → discover → backfill → download → extract → sync/verify.

The six stages (what each actually does)

1. Restore — c40_restore.py

Rehydrate a fresh/empty workspace from the newest ky_court-WIP-resume-*.tar.gz. Only acts when no DB exists; skips otherwise. If a stale workspace dir is present it is moved aside to -pre-restore (renamed, never deleted) before extracting — the "never destroys data" guarantee. Extracts to temp, then moves into place.

2. Discover — c50_discover.py

Enumerate the corpus by searching the court API so we know what exists. Builds search shards (YYYY-CA, YYYY-SC per year; ceiling-hit shards get sub-sharded YYYY-CA-0..9). Pages results with X-CTrack-Paging-* headers up to a 10k ceiling; a 400 or ceiling-cross raises to force sub-sharding rather than silently missing docs. Normalizes rows into documents with status='discovered', inserted idempotently via ON CONFLICT(document_id). Sweeps repeatedly until a pass adds fewer than CONVERGENCE_DELTA docs (convergence). Every sweep logged to discovery_runs.

3. Backfill — c60_backfill.py

Deduplicate existing backups into the content-addressed store. Hashes every file in pdf_backup/ (SHA-256), writes unique content to pdf_by_hash/<sha>.pdf (9,582 backup files → 3,326 unique PDFs) — "do we already have this?" becomes a hash lookup. Normalizes DB local_pdf pointers. Guarded by kv backfill_done.

4. Download — c70_download.py

Fetch PDFs for all discovered/failed docs with attempts < MAX_ATTEMPTS, in batches of 50. api_download does a HEAD (rejects non-200 and anything

MAX_PDF_BYTES), then streams GET, hashing as it goes. Dedup wins: a hash already in the store/manifest uploads nothing. Retry/backoff on transient errors: sleep = min(60, 5·2^attempts) (10→20→40→60s, capped), up to 4 attempts. Permanent errors (400/401/403/404/413) fail immediately, no backoff. Writes to tmp/<id>.part, promotes to store, paces itself, and pushes a heartbeat to status.json every 100 attempts.

5. Extract — c80_extract.py

Turn downloaded PDFs into searchable text with pypdf. Batches of 500 with 8 parallel worker threads. A PDF with no text layer is a valid "empty" result, not an error. Text written to text/<sha>.txt; extracted_chars recorded. Heuristic flags likely-scanned (image-only) PDFs when chars/pages < 20.

6. Sync/Verify — c90_verify.py

Prove nothing is corrupted and push an authoritative snapshot + summaries to Drive. Runs PRAGMA integrity_check; counts totals and the remaining queues (download + extract eligibility queries mirror the stage code exactly, so done is trustworthy). Exports document_inventory.csv + failed_inventory.csv. checkpoint() makes a consistent DB snapshot via src.backup(dst) (SQLite online backup API) and uploads it as the new ky_court.sqlite. Updates status.json with done = (queues empty). Prints PASS/FAIL.

Concurrency (the interesting part)

checkpoint() uses SQLite's online backup API (src.backup(dst)) from a separate connection — not copytree, not a WAL checkpoint. It copies the DB page-by-page and re-copies any page the writer modified during the pass, looping until a clean pass completes. Combined with WAL mode (readers don't block the writer) and busy_timeout=30000, this yields a transactionally-consistent snapshot even while Download is committing every doc. A naive file copy would produce a torn/corrupt snapshot and drop WAL-committed data.

Key design decisions

• Idempotent + resume-safe everywhere: skip what's done, never auto-delete on Drive.
• Named permanent-failure set (PERMANENT_STATUSES + _is_permanent()) distinguishes never-succeed (401/403/404/413) from transient (5xx/conn/429/408).
• Content-addressed store (pdf_by_hash/<sha>.pdf) makes dedup a lookup.
• Config via one CFG dict with env overrides (KY_BUDGET, KY_DRY, KY_PHASES, KY_RETRY_HARD, KY_LIVE).

Config knobs (top cell CFG)

Key	Default	Meaning
BUDGET_SECONDS	10h	max wall-clock per session (KY_BUDGET)
SHUTDOWN_SECONDS	15 min	stop early so the session finishes cleanly
PACING_SEARCH / PACING_DL	(0.3,0.7)/(0.3,0.5)	random sleep between API calls
MAX_ATTEMPTS	4	download retries before failed
MAX_PDF_BYTES	200 MiB	size cap; oversized (413) fails permanently, no retry/backoff
MAX_EXTRACT_ATTEMPTS	3	extraction retries before skip
RETRY_HARD	0 (KY_RETRY_HARD)	on resume, reset failed → discovered
PHASES	discover,backfill,download,extract	stages to run (KY_PHASES)
DRY_RUN	0 (KY_DRY)	1 = fully offline, no network

File layout

ky_court_pipeline.ipynb         the deliverable (generated)
notebook_builder/
  build.py                       regenerate the notebook from cells
  cells/  c00_config … c90_verify + drive_store.py
tests/  test_pipeline_local.py + conftest.py (offline; live gated by KY_LIVE)
scripts/  smoke_local.py         full offline pipeline over the real DB
RUNBOOK.md                      operational runbook (first run, resuming, failure modes)
docs/superpowers/ specs/ + plans/  design spec + implementation plan
PROJECT_NOTES.md                 this file
.gitignore                       ignores .venv, extracted/, *.sqlite, __pycache__,
                                 .worktrees/, .superpowers/  (NOT the WIP tars)

Data is NOT in the repo. extracted/, *.tar.gz resume/WIP archives, and sanitize-work/ live only in Drive / locally. Gitignore does not cover the WIP .tar.gz files, so never git add -A from the repo root.

Running / testing

python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
.venv/bin/pytest tests -q                # 20 offline tests (live ones skipped)
.venv/bin/python scripts/smoke_local.py  # full pipeline over the real 98,724-row DB
.venv/bin/python notebook_builder/build.py  # regenerate the notebook

Live API tests are gated behind KY_LIVE=1 and share a ≤10-request budget. Suite result at v1.2: 20 passed, 3 skipped.

Failure modes (from RUNBOOK)

• Colab disconnect mid-run → re-run; recover() heals stale rows, drops .part files.
• Drive quota exceeded → checkpoint push fails; run continues, store lags.
• 5xx/connection storms → backoff min(60, 5·2^attempts) up to MAX_ATTEMPTS.
• Search shard hits 10k ceiling → planner sub-shards next run.
• integrity_check not "ok" → DB corrupt; restore from last resume tar.

Versioning / releases

Immutable milestone tags (never force-move a published tag):

pipeline-v1  33a6ec1  original pipeline milestone
v1.1         ae77344  +README
v1.2         ec248fe  +413 oversize permanent-failure fix (MAX_PDF_BYTES knob)

v1.2 also has a GitHub release with changelog: https://github.com/stateofshit/kycourt-colab/releases/tag/v1.2

Status

Project complete and shipped. Suite green offline. The pipeline is ready to run in Colab (restore from the resume tar → Run all).

Roadmap / Future enhancements

• Parallel extract workers — tune the 8-worker batch size based on observed Colab runtime limits.
• Drive quota monitoring — automated alert when status.json reports approaching quota, with graceful degradation to skip upload.
• CLI entry point — kycourt-pipeline run --phase discover to run individual stages outside Colab.
• Query layer — lightweight read-only API over the SQLite DB for inspecting discovered/extracted docs without re-running the pipeline.
• Vault migration — if/when the repo is folded into the vault, adapt folder frontmatter and consolidate .tar.gz WIP archives under the vault's resume format. Remote backup, README, tags, and v1.2 release all in place on GitHub. Suite green offline. The pipeline is ready to run in Colab (restore from the resume tar → Run all).