8.6 KiB
title, status, folder, tags, created, updated, version
| title | status | folder | tags | created | updated | version | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| KY Courts Archive Pipeline | active | 02-notes |
|
2026-08-12 | 2026-08-12 | 1.0.0 |
KY Courts Archive Pipeline
A Colab-based harvester that builds a local, full-library archive of Kentucky Court of Appeals (CA) and Supreme Court (SC) filings from the state's public C-Track document portal, and persists everything to Google Drive so long runs survive VM tears.
Repo: git@github.com:stateofshit/ky-court-archive-pipeline.git (private, pushed as origin/main)
Deliverable: KY_Courts_Archive_Pipeline.ipynb (a 12-cell Colab notebook) + colab-cell-magic-reference.md
Why this project exists
The KY judiciary serves case documents through appellatepublic.kycourts.net — a public C-Track search API. There is no bulk download, only paginated search. To archive the whole library (roughly 4–6 hours / ~3,900 pages of API calls across ~70 shards), you need:
- A way to enumerate every document without blowing Colab's session limits.
- Resume + checkpointing so a killed session doesn't restart from page 1.
- A durable copy on Drive so a torn-down VM loses nothing.
- A completeness check that tells you what you'd actually be missing if you stopped.
That's the whole job. Nothing here is a toy — it ends in a verified, resumable local archive.
Source API
- Search:
https://appellatepublic.kycourts.net/api/api/v1/publicaccessdocuments/search - Download:
https://appellatepublic.kycourts.net/api/api/v1/publicaccessdocuments/{id}/download - Auth: none — public. Only a
User-Agentheader is set. - Paging: via
X-CTrack-Paging-StartIndex/-MaxResultsheaders (max practical page = 25). - Facets: returned on page 1 per shard and used to estimate total documents (drives the ceiling guard and ETA).
Seen live in cell 5 (CTrackClient, exponential-backoff retries, connection pooling tuned for Stage B's 10 threads).
Notebook map (cells run top-to-bottom)
| # | Title | What it does |
|---|---|---|
| 1 | Install Dependencies | pip install + imports |
| 2 | Pipeline Configuration | PipelineConfig dataclass → CONFIG |
| 3 | Mount Drive + dirs | Drive auth, create local + Drive trees, logging |
| 4 | DB Schema + Restore | create tables, auto-restore latest Drive snapshot |
| 5 | C-Track API Client | retry/paging/download client |
| 6 | Parsing, Classification & Job Enqueue | formatting, keyword filters, discovery_jobs seeding |
| 7 | State Manager & Pre-flight | HarvesterState, per-shard ETA table |
| 8 | Stage A — Shard harvest loop | the long metadata harvest |
| 9 | Completeness Report | flags capped/failed/pending/under-covered shards |
| 10 | Stage B — PDF download | parallelized, resumable downloads |
| 11 | Status Dashboard & Export | counts + exports |
Data model (sqlite, ky_court.sqlite)
Key tables created in cell 4:
discovery_jobs— one row per case-prefix shard; tracksstatus,start_index(page cursor),pages_done,extra_json(holds the auto-retryattemptscounter),last_error.documents— deduped bydocument_id; stores case number, type, filing date, classification (current_decision,decision_confidence).document_discovery— which job discovered which doc (idempotent join).download_queue/download_attempts/files/file_duplicates— PDF download state machine + the downloaded artifacts.text_extractions/text_chunks/fts_chunks— OCR/text layer (FTS5) for search.api_queries,errors,field_probes,filter_decisions,sync_events— audit + diagnostics.
documents upsert is idempotent: INSERT ... ON CONFLICT(document_id) DO UPDATE, so re-fetching overlapping pages never creates duplicates.
Core design patterns
1. Shard-based harvesting
The library is split into case-prefix shards by year × court (e.g. 1996-CA-, 2005-SC-). Stage A walks each shard's pages until the API returns two consecutive empty pages.
2. Sub-sharding (complete-library support)
If a year-court shard contains > sub_shard_threshold (9000) docs, it's split recursively by appending case-number digits (2020-CA-1, 2020-CA-15, …) up to sub_shard_max_depth (4). Keeps every shard small enough to paginate reliably.
3. 10k ceiling guard
If the facet estimate for a shard exceeds 10,000, the C-Track API rejects deep pagination. Stage A marks such shards ceiling (not done) so the completeness report explicitly flags them as not harvested rather than silently skipping.
4. Per-page checkpoints
Every page commits start_index (the next cursor) and pages_done to discovery_jobs immediately. A disconnect mid-shard resumes from the last committed page — never from page 1.
5. Drive snapshots
safe_snapshot() backs up the full DB to MyDrive/ky_court-WIP/db_snapshots/ky_court_<ts>.sqlite every N docs / N seconds and at end of Stage A. Cell 4 auto-restores the latest snapshot on the next session → the run continues where it left off.
6. Session chunking (stage_a_from / stage_a_to)
The full sorted shard list is split into per-session index ranges so one long run can be fanned out across multiple Colab sessions ((0,20), (21,40), …). Indexes reference the full sorted list, so positions stay stable across sessions as shards complete. Default (0,-1) = everything.
7. Attempt-capped auto-retry
A failed shard is auto-included in the next run while its attempts (stored in extra_json) is < max_shard_attempts (default 2). Once it hits the cap it drops out of auto-retry permanently and is surfaced by the Completeness Report until manually reset:
con.execute("UPDATE discovery_jobs SET status='queued', extra_json='{}' WHERE status='failed'")
con.commit()
This self-heals transient failures without letting a genuinely broken shard burn session time forever.
8. Failure states
done— harvested. (But the report cross-checks: done but 0 docs in DB → flag.)ceiling— capped at 10k, docs NOT harvested → needs attention.failed— threw after retries; auto-retried until the attempt cap.queued/running/superseded— normal lifecycle / replaced by a sub-shard.
Configuration knobs (PipelineConfig, cell 2)
| Knob | Default | Purpose |
|---|---|---|
dry_run |
True |
True = sample first shard only; set False for real harvest |
year_start / year_end |
1996 / 2026 | harvest scope |
courts |
('CA','SC') |
Counties Appeals + Supreme |
sub_shard_threshold |
9000 | auto sub-shard threshold |
sub_shard_max_depth |
4 | sub-shard depth |
stage_a_from / stage_a_to |
0 / -1 | session chunk range |
max_shard_attempts |
2 | auto-retry cap |
request_delay_seconds |
0.4 | politeness / rate-limit delay |
max_retries |
4 | per-request backoff |
download_included |
False |
flip True to start PDF downloads (Stage B) |
drive_wip |
/content/drive/MyDrive/ky_court-WIP |
Drive working folder |
How to run (Colab)
- Upload
KY_Courts_Archive_Pipeline.ipynb; use a CPU runtime. - Run cells 1→11 in order. Cell 3 triggers Drive auth.
- In cell 2: set
dry_run=Falsewhen ready for a real run; set the chunk range for session-split runs. - For every new session, run 1–4 first (auto-restores the DB), set the chunk, then Stage A.
- After Stage A, run the Completeness Report (cell 9) — the same Google account / same
ky_court-WIPfolder must be used across sessions so snapshots land in one place.
Full setup checklist is in the Colab prompt I issued from this note's project work — the key gotchas are: same Drive account every session, cells in order, and dry_run=False before Stage A.
Status & session history
- 2026-08-12 — added session chunking (
stage_a_from/stage_a_to) and attempt-capped auto-retry (max_shard_attempts=2, counter inextra_json, Completeness Report showsattempts a/max). - Prior: full-range harvest notebook, ID parsing/classification, 10k ceiling guard, auto sub-sharding, resumable checkpoints, Drive snapshots, Stage B downloads, FTS search.
Failure modes to keep in mind
- A
failed-state shard won't auto-resume on a plain re-run after it hits the cap — reset it explicitly (pattern above) or it silently skips. This is the known gotcha the attempt-cap is designed around: it's bounded effort by design, not a bug. - Drive account mismatch across sessions breaks the snapshot/resume trail.
- Deep pagination >10k is rejected by the API — that's exactly what the
ceilingguard exists to surface.