---
title: "kycourt-colab — Kentucky Appellate Court Corpus Pipeline (project notes)"
status: "active"
folder: "repo root (kycourt-colab)"
tags: [kycourt, pipeline, colab, project-notes]
created: "2026-08-12"
updated: "2026-08-12"
version: "1.1.0"
---

# kycourt-colab — Kentucky Appellate Court Corpus Pipeline

## Overview

A single **generated Google Colab notebook** (`ky_court_pipeline.ipynb`) that
restores, discovers, downloads, extracts, and verifies the full Kentucky
appellate court document corpus (~**98.7k documents**). It is built to survive
Colab's ephemeral runtimes: it auto-resumes after disconnects, is idempotent,
and mirrors all state to Google Drive so nothing is lost between sessions.

**Remote:** https://github.com/stateofshit/kycourt-colab (private)
**Local worktree:** `/home/user/00-incoming/kycourt-colab`

## What problem it solves

A batch archival crawl of public court documents over a ~10-hour-per-session
horizon, on infrastructure (Colab) that kills your process mid-run. Requirements
that fall out of that:

- Every session must pick up exactly where the last one stopped.
- Re-running must be safe (idempotent) — no double-downloads, no data loss.
- State must live somewhere durable (Drive), because the runtime is throwaway.
- The crawl must be polite to a public API (paced, backoff, retries) and
  deduplicate aggressively (content-addressed storage).

## Architecture

The notebook is **not hand-written** — its cells are inlined from real Python
modules in `notebook_builder/cells/` (one module per cell, `c00` → `c90`), and
regenerated with `notebook_builder/build.py`. This means the pipeline is
version-controlled, unit-testable Python that *just happens to be shipped as* a
Colab notebook.

Two shared abstractions carry every stage:

- **SQLite DB** (`ky_court.sqlite`) — source of truth: `documents`,
  `discovery_runs`, `extraction_log`, `kv`. Opened in **WAL mode** with a
  30s `busy_timeout` so a background reader can snapshot it while the writer
  commits (see concurrency below).
- **`Store` abstraction** — Google Drive in Colab, local filesystem in tests;
  exposes `exists / read_file / write_bytes / upload_file / list_dir`. All
  stages talk to it, never to Drive directly.

**Stage order:** `restore → discover → backfill → download → extract → sync/verify`.

## The six stages (what each actually does)

### 1. Restore — `c40_restore.py`
Rehydrate a fresh/empty workspace from the newest `ky_court-WIP-resume-*.tar.gz`.
Only acts when no DB exists; skips otherwise. If a stale workspace dir is
present it is **moved aside to `-pre-restore` (renamed, never deleted)** before
extracting — the "never destroys data" guarantee. Extracts to temp, then moves
into place.

### 2. Discover — `c50_discover.py`
Enumerate the corpus by searching the court API so we know *what exists*.
Builds search shards (`YYYY-CA`, `YYYY-SC` per year; ceiling-hit shards get
sub-sharded `YYYY-CA-0..9`). Pages results with `X-CTrack-Paging-*` headers up
to a 10k ceiling; a `400` or ceiling-cross raises to force sub-sharding rather
than silently missing docs. Normalizes rows into `documents` with
`status='discovered'`, inserted idempotently via `ON CONFLICT(document_id)`.
Sweeps repeatedly until a pass adds fewer than `CONVERGENCE_DELTA` docs
(convergence). Every sweep logged to `discovery_runs`.

### 3. Backfill — `c60_backfill.py`
Deduplicate *existing* backups into the content-addressed store. Hashes every
file in `pdf_backup/` (SHA-256), writes unique content to `pdf_by_hash/<sha>.pdf`
(9,582 backup files → 3,326 unique PDFs) — "do we already have this?" becomes a
hash lookup. Normalizes DB `local_pdf` pointers. Guarded by `kv backfill_done`.

### 4. Download — `c70_download.py`
Fetch PDFs for all `discovered`/`failed` docs with `attempts < MAX_ATTEMPTS`,
in batches of 50. `api_download` does a `HEAD` (rejects non-200 and anything
> `MAX_PDF_BYTES`), then streams `GET`, hashing as it goes. Dedup wins: a hash
already in the store/manifest uploads nothing. **Retry/backoff** on transient
errors: `sleep = min(60, 5·2^attempts)` (10→20→40→60s, capped), up to 4
attempts. Permanent errors (400/401/403/404/**413**) fail immediately, no
backoff. Writes to `tmp/<id>.part`, promotes to store, paces itself, and pushes
a heartbeat to `status.json` every 100 attempts.

### 5. Extract — `c80_extract.py`
Turn downloaded PDFs into searchable text with `pypdf`. Batches of 500 with 8
parallel worker threads. A PDF with no text layer is a valid `"empty"` result,
not an error. Text written to `text/<sha>.txt`; `extracted_chars` recorded.
Heuristic flags likely-scanned (image-only) PDFs when `chars/pages < 20`.

### 6. Sync/Verify — `c90_verify.py`
Prove nothing is corrupted and push an authoritative snapshot + summaries to
Drive. Runs `PRAGMA integrity_check`; counts totals and the **remaining
queues** (download + extract eligibility queries mirror the stage code exactly,
so `done` is trustworthy). Exports `document_inventory.csv` +
`failed_inventory.csv`. `checkpoint()` makes a consistent DB snapshot via
`src.backup(dst)` (SQLite online backup API) and uploads it as the new
`ky_court.sqlite`. Updates `status.json` with `done = (queues empty)`.
Prints `PASS`/`FAIL`.

## Concurrency (the interesting part)

`checkpoint()` uses SQLite's **online backup API** (`src.backup(dst)`) from a
**separate connection** — not `copytree`, not a WAL checkpoint. It copies the
DB page-by-page and re-copies any page the writer modified during the pass,
looping until a clean pass completes. Combined with **WAL mode** (readers don't
block the writer) and **`busy_timeout=30000`**, this yields a
transactionally-consistent snapshot even while Download is committing every doc.
A naive file copy would produce a torn/corrupt snapshot and drop WAL-committed
data.

## Key design decisions

- **Idempotent + resume-safe everywhere**: skip what's done, never auto-delete
  on Drive.
- **Named permanent-failure set** (`PERMANENT_STATUSES` + `_is_permanent()`)
  distinguishes never-succeed (401/403/404/413) from transient (5xx/conn/429/408).
- **Content-addressed store** (`pdf_by_hash/<sha>.pdf`) makes dedup a lookup.
- **Config via one `CFG` dict** with env overrides (`KY_BUDGET`, `KY_DRY`,
  `KY_PHASES`, `KY_RETRY_HARD`, `KY_LIVE`).

## Config knobs (top cell `CFG`)

| Key | Default | Meaning |
|---|---|---|
| `BUDGET_SECONDS` | 10h | max wall-clock per session (`KY_BUDGET`) |
| `SHUTDOWN_SECONDS` | 15 min | stop early so the session finishes cleanly |
| `PACING_SEARCH` / `PACING_DL` | (0.3,0.7)/(0.3,0.5) | random sleep between API calls |
| `MAX_ATTEMPTS` | 4 | download retries before `failed` |
| `MAX_PDF_BYTES` | 200 MiB | size cap; oversized (413) fails permanently, no retry/backoff |
| `MAX_EXTRACT_ATTEMPTS` | 3 | extraction retries before skip |
| `RETRY_HARD` | 0 (`KY_RETRY_HARD`) | on resume, reset `failed` → `discovered` |
| `PHASES` | discover,backfill,download,extract | stages to run (`KY_PHASES`) |
| `DRY_RUN` | 0 (`KY_DRY`) | 1 = fully offline, no network |

## File layout

```
ky_court_pipeline.ipynb         the deliverable (generated)
notebook_builder/
  build.py                       regenerate the notebook from cells
  cells/  c00_config … c90_verify + drive_store.py
tests/  test_pipeline_local.py + conftest.py (offline; live gated by KY_LIVE)
scripts/  smoke_local.py         full offline pipeline over the real DB
RUNBOOK.md                      operational runbook (first run, resuming, failure modes)
docs/superpowers/ specs/ + plans/  design spec + implementation plan
PROJECT_NOTES.md                 this file
.gitignore                       ignores .venv, extracted/, *.sqlite, __pycache__,
                                 .worktrees/, .superpowers/  (NOT the WIP tars)
```

> **Data is NOT in the repo.** `extracted/`, `*.tar.gz` resume/WIP archives, and
> `sanitize-work/` live only in Drive / locally. Gitignore does not cover the
> WIP `.tar.gz` files, so never `git add -A` from the repo root.

## Running / testing

```bash
python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
.venv/bin/pytest tests -q                # 20 offline tests (live ones skipped)
.venv/bin/python scripts/smoke_local.py  # full pipeline over the real 98,724-row DB
.venv/bin/python notebook_builder/build.py  # regenerate the notebook
```

Live API tests are gated behind `KY_LIVE=1` and share a ≤10-request budget.
Suite result at v1.2: **20 passed, 3 skipped**.

## Failure modes (from RUNBOOK)

- **Colab disconnect mid-run** → re-run; `recover()` heals stale rows, drops
  `.part` files.
- **Drive quota exceeded** → checkpoint push fails; run continues, store lags.
- **5xx/connection storms** → backoff `min(60, 5·2^attempts)` up to `MAX_ATTEMPTS`.
- **Search shard hits 10k ceiling** → planner sub-shards next run.
- **`integrity_check` not "ok"** → DB corrupt; restore from last resume tar.

## Versioning / releases

Immutable milestone tags (never force-move a published tag):

```
pipeline-v1  33a6ec1  original pipeline milestone
v1.1         ae77344  +README
v1.2         ec248fe  +413 oversize permanent-failure fix (MAX_PDF_BYTES knob)
```

`v1.2` also has a **GitHub release** with changelog:
https://github.com/stateofshit/kycourt-colab/releases/tag/v1.2

## Status

Project complete and shipped. Suite green offline. The pipeline is ready to run in Colab (restore from the resume tar → Run all).

### Roadmap / Future enhancements
- **Parallel extract workers** — tune the 8-worker batch size based on observed Colab runtime limits.
- **Drive quota monitoring** — automated alert when `status.json` reports approaching quota, with graceful degradation to skip upload.
- **CLI entry point** — `kycourt-pipeline run --phase discover` to run individual stages outside Colab.
- **Query layer** — lightweight read-only API over the SQLite DB for inspecting discovered/extracted docs without re-running the pipeline.
- **Vault migration** — if/when the repo is folded into the vault, adapt `folder` frontmatter and consolidate `.tar.gz` WIP archives under the vault's resume format. Remote backup, README, tags, and v1.2 release all
in place on GitHub. Suite green offline. The pipeline is ready to run in Colab
(restore from the resume tar → Run all).