vault backup: 2026-08-12 15:38:55

2026-08-12 15:38:55 -04:00
parent 594346c495
commit 22c31d7bd4
23 changed files with 1255 additions and 155 deletions
@@ -0,0 +1,527 @@
+# 🔗 AI Handoff — Projects & Agent Workspace
+
+**Live at:** [https://app.ai-handoff.work](https://app.ai-handoff.work)  
+**Repo:** [https://github.com/stateofshit/url-shortener](https://github.com/stateofshit/url-shortener)  
+**Built:** August 12, 2026
+
+* * *
+
+## What This Covers
+
+### 1. URL Shortener (Production App)
+A **production-ready URL shortener** with **live click analytics**, deployed entirely on **Cloudflare's edge infrastructure**. No accounts, no databases to manage, no servers to maintain. Just paste a long URL, get a short one, and watch the clicks roll in.
+
+📄 See below for full URL Shortener documentation.
+
+### 2. Assistant Workspace (Agent Project)
+An **AI agent starter** built with Cloudflare Agents SDK + TypeScript, living at:
+
+```
+/home/user/03-projects/assistant-workspace/
+```
+
+Copied from `.agents/skills/agents-sdk/my-agent/`. Purpose-built for autonomous agent behavior using the Cloudflare Workers platform.
+
+#### Quick Start
+```bash
+cd /home/user/03-projects/assistant-workspace
+npx wrangler dev    # local dev
+npx wrangler deploy  # production
+```
+
+#### Structure
+```
+assistant-workspace/
+├── AGENTS.md       # Agent guidelines & docs
+├── src/            # Agent source code
+├── test/           # Vitest tests
+├── wrangler.jsonc  # Worker bindings config
+└── package.json    # Agents SDK deps
+```
+
+* * *
+
+## 🏗️ Architecture Overview (URL Shortener)
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                         YOUR BROWSER                            │
+└─────────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────────┐
+│                    Cloudflare DNS (ai-handoff.work)             │
+│  shit.ai-handoff.work  →  Worker API                           │
+│  dash.ai-handoff.work  →  Dashboard (Pages)                    │
+│  app.ai-handoff.work   →  React App (Pages)                    │
+└─────────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────────┐
+│              Cloudflare Workers + Pages + D1                    │
+│                                                                 │
+│  ┌─────────────────────────────────────────────────────────┐   │
+│  │  Worker: url-shortener-api (shit.ai-handoff.work)      │   │
+│  │  - POST /api/shorten                                    │   │
+│  │  - GET  /api/links                                      │   │
+│  │  - GET  /api/trends/:code                               │   │
+│  │  - GET  /{code}  → 301 redirect to original URL         │   │
+│  └─────────────────────────────────────────────────────────┘   │
+│                                                                 │
+│  ┌─────────────────────────────────────────────────────────┐   │
+│  │  Pages: shrtlink (dash.ai-handoff.work)                 │   │
+│  │  - Static HTML/CSS/JS dashboard with live stats         │   │
+│  │  - Copy-link button, 14-day sparkline trends            │   │
+│  └─────────────────────────────────────────────────────────┘   │
+│                                                                 │
+│  ┌─────────────────────────────────────────────────────────┐   │
+│  │  Pages: shrtlink-app (app.ai-handoff.work)              │   │
+│  │  - React Vite app (same functionality, different UI)    │   │
+│  │  - Built with React 19 + Vite 6                         │   │
+│  └─────────────────────────────────────────────────────────┘   │
+│                                                                 │
+│  ┌─────────────────────────────────────────────────────────┐   │
+│  │  D1 Database: url-shortener-db                          │   │
+│  │  - urls table: code, url, clicks, created_at            │   │
+│  │  - clicks table: code, clicked_at (per-click history)   │   │
+│  └─────────────────────────────────────────────────────────┘   │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+* * *
+
+## 🛠️ Tech Stack (URL Shortener)
+
+| Layer | Technology | Purpose |
+| --- | --- | --- |
+| **Edge Runtime** | Cloudflare Workers | URL shortening, redirects, API endpoints |
+| **Static Sites** | Cloudflare Pages | Dashboard + React app deployment |
+| **Database** | Cloudflare D1 (SQLite) | Store URLs, click counts, click timestamps |
+| **Frontend (Dashboard)** | Vanilla HTML/CSS/JS | Lightweight, fast, no build step |
+| **Frontend (React App)** | React 19 + Vite 6 | Modern SPA with same features |
+| **CI/CD** | GitHub Actions | Auto-deploy on `git push master` |
+| **Custom Domains** | Cloudflare DNS | `*.ai-handoff.work` |
+
+**Agent Workspace uses:** Cloudflare Workers + Agents SDK + TypeScript + Vitest
+
+* * *
+
+## 🌐 Custom Domains (URL Shortener)
+
+All domains point to Cloudflare infrastructure and are **proxied** (CDN + security enabled):
+| Domain | Target | Status |
+| --- | --- | --- |
+| `shit.ai-handoff.work` | Worker API | ✅ Live (200) |
+| `dash.ai-handoff.work` | Dashboard (Pages `shrtlink`) | ✅ Live (200) |
+| `app.ai-handoff.work` | React App (Pages `shrtlink-app`) | ✅ Live (200) |
+
+**Why custom domains?**
+
+*   Professional URLs (not `workers.dev` or `pages.dev`)
+*   SSL certificates auto-provisioned by Cloudflare
+*   CDN caching at the edge
+*   DDoS protection via Cloudflare
+
+* * *
+
+## ✨ Features (URL Shortener)
+
+### 1. URL Shortening
+*   Paste any `http://` or `https://` URL
+*   Get a 6-character short code (e.g., `f5RHkl`)
+*   Short URL: `https://shit.ai-handoff.work/f5RHkl`
+*   **Strict validation**: rejects `javascript:`, `ftp://`, etc.
+*   **Dedupe**: shortening the same URL twice returns the existing code
+
+### 2. Live Dashboard
+*   **Total links count** (real-time from D1)
+*   **Total clicks count** (sum of all link clicks)
+*   **Table of all links** with:
+  *   Short URL (clickable)
+  *   Destination URL (truncated if >60 chars)
+  *   Click count (badge)
+  *   14-day trend sparkline (SVG bar chart)
+  *   Creation date
+  *   Copy button (⎘ → ✓ with clipboard API)
+*   **Auto-refresh** every 15 seconds
+
+### 3. Click Analytics
+*   **Per-click tracking**: every redirect logs a timestamp to `clicks` table
+*   **Daily buckets**: `/api/trends/:code` returns `{day, n}` for last 14 days
+*   **Sparkline visualization**: inline SVG bar chart per link
+*   **Zero data loss**: historical clicks preserved even if URL is deleted
+
+### 4. Developer Experience
+*   **Auto-deploy**: push to `master` → GitHub Actions → Cloudflare
+*   **Local dev**: `npm run dev` (Vite) + `wrangler dev` (Worker)
+*   **Type-safe**: D1 queries via Wrangler bindings
+*   **Testable**: Playwright E2E tests in `/tmp/*.mjs`
+
+* * *
+
+## 📁 Project Structure (URL Shortener)
+
+```
+url-shortener/
+├── .github/workflows/
+│   └── deploy.yml              # GitHub Actions CI/CD
+├── .wrangler/                  # Wrangler cache (gitignored)
+├── dashboard/                  # Static dashboard (HTML/CSS/JS)
+│   ├── index.html
+│   ├── app.js                  # Dashboard logic
+│   └── styles.css              # Dark theme styles
+├── dist/                       # Vite build output (gitignored)
+├── src/                        # React app source (Vite)
+│   ├── main.jsx
+│   └── App.jsx
+├── wrangler-api/               # Worker + D1 migrations
+│   ├── src/
+│   │   └── index.js            # Worker fetch handler
+│   ├── migrations/
+│   │   ├── 000_create_urls.sql
+│   │   ├── 001_add_clicks.sql
+│   │   └── 002_create_clicks.sql
+│   ├── wrangler.jsonc          # Worker config + D1 binding
+│   └── package.json
+├── .env                        # Cloudflare credentials (gitignored)
+├── .env.example                # Example env vars
+├── .gitignore
+├── index.html                  # Root (redirects to dashboard)
+├── package.json                # Vite + React deps
+└── vite.config.js
+```
+
+* * *
+
+## 🗄️ Database Schema (URL Shortener - D1)
+
+### `urls` table
+
+```sql
+CREATE TABLE urls (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  code TEXT NOT NULL UNIQUE,           -- 6-char short code
+  url TEXT NOT NULL,                   -- original URL
+  clicks INTEGER NOT NULL DEFAULT 0,   -- running total
+  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### `clicks` table (migration 002)
+
+```sql
+CREATE TABLE clicks (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  code TEXT NOT NULL,
+  clicked_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+  FOREIGN KEY (code) REFERENCES urls(code)
+);
+CREATE INDEX idx_clicks_code ON clicks(code, clicked_at);
+```
+
+**Why two tables?**
+
+*   `urls.clicks` = fast counter for the dashboard badge
+*   `clicks` table = per-click history for trend charts
+
+* * *
+
+## 🔌 API Endpoints (URL Shortener)
+
+All endpoints on `https://shit.ai-handoff.work`:
+
+### `POST /api/shorten`
+
+**Request:**
+
+```json
+{ "url": "https://example.com/very/long/path" }
+```
+
+**Response (200):**
+
+```json
+{
+  "shortUrl": "https://shit.ai-handoff.work/f5RHkl",
+  "code": "f5RHkl",
+  "dedupe": false
+}
+```
+
+**Errors:**
+*   `400` — invalid URL, missing `url` field
+*   `500` — internal error
+
+### `GET /api/links`
+
+**Response (200):**
+
+```json
+{
+  "links": [
+    {
+      "code": "f5RHkl",
+      "url": "https://example.com/long/path",
+      "clicks": 42,
+      "created_at": "2026-08-12T03:00:00Z"
+    }
+  ]
+}
+```
+
+### `GET /api/trends/:code`
+
+**Response (200):**
+
+```json
+{
+  "code": "f5RHkl",
+  "daily": [
+    { "day": "2026-08-11", "n": 3 },
+    { "day": "2026-08-12", "n": 4 }
+  ]
+}
+```
+
+### `GET /{code}` (redirect)
+
+**Behavior:**
+*   301 Moved Permanently → original URL
+*   Increments `urls.clicks` + logs to `clicks` table
+*   CORS headers for cross-origin requests
+
+* * *
+
+## 🚀 Deployment
+
+### Manual Deploy (URL Shortener)
+
+```bash
+# Worker
+cd wrangler-api && npx wrangler deploy
+
+# Dashboard (Pages)
+cd dashboard && npx wrangler pages deploy ./ --project-name=shrtlink
+
+# React app (Pages)
+npx wrangler pages deploy ./dist --project-name=shrtlink-app
+```
+
+### Auto-Deploy (GitHub Actions)
+
+```yaml
+# Triggered on: git push origin master
+# Steps:
+# 1. Build Vite app (npm run build)
+# 2. Deploy Worker (npx wrangler deploy)
+# 3. Deploy Dashboard (npx wrangler pages deploy)
+# 4. Deploy React app (npx wrangler pages deploy)
+```
+
+**Secrets required:**
+*   `CLOUDFLARE_API_TOKEN` — Cloudflare API token with Pages + Workers permissions
+*   `CLOUDFLARE_ACCOUNT_ID` — Cloudflare account ID
+
+### Agent Workspace Deploy
+
+```bash
+cd /home/user/03-projects/assistant-workspace
+npx wrangler deploy
+```
+
+* * *
+
+## 🧪 Testing (URL Shortener)
+
+### Local Development
+
+```bash
+# Terminal 1: Worker (localhost:8787)
+cd wrangler-api && npx wrangler dev
+
+# Terminal 2: Vite dev server (localhost:5173)
+npm run dev
+
+# Terminal 3: Dashboard (localhost:3000)
+cd dashboard && npx wrangler pages dev ./
+```
+
+### E2E Tests (Playwright)
+
+```bash
+# Test dashboard features
+node /tmp/dash-features.mjs
+
+# Test copy button + trends
+node /tmp/copy-test.mjs
+
+# Full end-to-end
+node /tmp/custom-e2e.mjs
+```
+
+### Agent Workspace Tests
+
+```bash
+cd /home/user/03-projects/assistant-workspace
+npx vitest
+```
+
+* * *
+
+## 🔒 Security (URL Shortener)
+
+*   **Strict URL validation**: only `http:` and `https:` protocols allowed
+*   **CORS**: `Access-Control-Allow-Origin: *` (public API)
+*   **Dedupe**: prevents duplicate rows for same URL
+*   **Collision check**: 6-char codes from 62^6 = ~56B combinations
+*   **SSL**: auto-provisioned by Cloudflare for all custom domains
+*   **DDoS protection**: Cloudflare edge network
+
+* * *
+
+## 📊 Live Stats (URL Shortener — as of Aug 12, 2026)
+
+| Metric | Value |
+| --- | --- |
+| Total links | 11 |
+| Total clicks | 8 |
+| Active domains | 3 |
+| Deployments | 42s avg |
+| Uptime | 100% |
+
+* * *
+
+## 🎯 How to Use
+
+### As a User (URL Shortener)
+1.  Go to [**https://app.ai-handoff.work**](https://app.ai-handoff.work) or [**https://dash.ai-handoff.work**](https://dash.ai-handoff.work)
+2.  Paste a long URL into the input field
+3.  Click **Shorten**
+4.  Copy the short URL (⎘ button) or click to open it
+5.  Share it — when people click it, you'll see the click count increase in real-time
+
+### As a Developer (URL Shortener)
+1.  Clone the repo: `git clone https://github.com/stateofshit/url-shortener`
+2.  Set up Cloudflare credentials in `.env` (see `.env.example`)
+3.  Run locally: `npm run dev` + `cd wrangler-api && npx wrangler dev`
+4.  Deploy: `git push origin master` (auto-deploys via GitHub Actions)
+
+### As a Developer (Agent Workspace)
+1.  Workspace lives at: `/home/user/03-projects/assistant-workspace/`
+2.  Source synced from: `.agents/skills/agents-sdk/my-agent/`
+3.  Run locally: `cd assistant-workspace && npx wrangler dev`
+4.  Deploy: `cd assistant-workspace && npx wrangler deploy`
+
+* * *
+
+## Key Design Decisions (URL Shortener)
+
+### Why Cloudflare?
+*   **Edge runtime**: Workers execute at the edge (closest to user)
+*   **Free tier**: 100K requests/day on Workers, 10K on D1
+*   **Zero config**: No servers, no scaling, no DevOps
+*   **Built-in SSL**: Automatic HTTPS for custom domains
+
+### Why D1 (SQLite) instead of KV?
+*   **SQL queries**: need `GROUP BY`, `ORDER BY`, `JOIN` for analytics
+*   **Relational**: `clicks` table references `urls` table
+*   **Cost**: D1 is cheaper than Vectorize for this use case
+
+### Why two frontends?
+*   **Dashboard** (vanilla): lightweight, fast, no build step
+*   **React app** (Vite): modern SPA, same features, different UI
+*   **Proof of concept**: shows both approaches work on Cloudflare
+
+### Why 6-character codes?
+*   **Short**: `f5RHkl` is memorable
+*   **Collision-resistant**: 62^6 = 56 billion combinations
+*   **URL-friendly**: no special characters, no encoding needed
+
+* * *
+
+## 🚧 Known Limitations (URL Shortener)
+
+1.  **No user accounts**: anyone with the link can see the dashboard
+2.  **No custom codes**: codes are randomly generated (no `bit.ly`-style custom slugs)
+3.  **No link expiration**: links never expire (can add TTL later)
+4.  **No QR codes**: short URLs are text only (can add QR generation)
+5.  **No mobile app**: web-only (can add PWA later)
+
+* * *
+
+## 🔮 Future Enhancements
+
+### High Priority (URL Shortener)
+- [ ] **Cloudflare Access**: lock `dash.ai-handoff.work` behind login
+- [ ] **Custom codes**: allow users to set their own short URL
+- [ ] **Link groups**: organize links into campaigns
+- [ ] **Export data**: download click analytics as CSV/JSON
+
+### Medium Priority (URL Shortener)
+- [ ] **QR codes**: generate QR for each short URL
+- [ ] **UTM tracking**: append `?utm_source=twitter` to destination
+- [ ] **Link expiration**: set TTL
+- [ ] **Click geo**: log country/region per click
+
+### Low Priority (URL Shortener)
+- [ ] **Mobile PWA**: installable app with offline support
+- [ ] **Browser extension**: right-click → "Shorten this URL"
+- [ ] **API keys**: rate-limit per user
+- [ ] **Webhooks**: notify on new click
+
+* * *
+
+## 📝 Git History (Key Commits — URL Shortener)
+
+| Commit | Message | Date |
+| --- | --- | --- |
+| `ab16107` | feat: strict URL validation, dedupe, click analytics; add landing page + live dashboard | Aug 11 |
+| `c98074` | chore: point frontends at custom API domain shit.ai-handoff.work | Aug 12 |
+| `c66bd15` | chore: deploy vite app as shrtlink-app Pages project on app.ai-handoff.work | Aug 12 |
+| `48481af` | feat: add copy-link button + 14-day click trend sparklines to dashboard | Aug 12 |
+| `0562e49` | ci: use npx wrangler for all deployments | Aug 12 |
+
+* * *
+
+## 💡 Lessons Learned (URL Shortener)
+
+1.  **Custom domains need DNS records**: Cloudflare Pages doesn't auto-create CNAME for all subdomains — had to manually add `app → shrtlink-app.pages.dev`
+2.  **D1 migrations are idempotent**: `CREATE TABLE IF NOT EXISTS` is safe to run multiple times
+3.  **Click logging is cheap**: D1 is SQLite, so logging every click is fine
+4.  **Sparklines are easy**: inline SVG with `<rect>` elements, no chart library needed
+5.  **GitHub Actions secrets are mandatory**: workflow fails silently if secrets are missing
+6.  **Wrangler CLI is powerful**: `npx wrangler pages deploy` works from any directory
+
+## 💡 Lessons Learned (Agent Workspace)
+
+1.  **Agent skills stored in `.agents/skills/`** — the canonical source for agent templates
+2.  **Working copies go in `03-projects/`** — keeps active projects organized alongside other work
+3.  **Sync when relevant** — changes to upstream agent skills should be reflected in the working copy
+
+* * *
+
+## 🎉 Summary
+
+### URL Shortener
+This is a **fully functional, production-ready URL shortener** that:
+*   ✅ Shortens URLs with 6-character codes
+*   ✅ Tracks every click with timestamps
+*   ✅ Shows live analytics with sparkline trends
+*   ✅ Deploys automatically on `git push`
+*   ✅ Runs on Cloudflare's edge (fast, free, scalable)
+*   ✅ Uses custom domains (`*.ai-handoff.work`)
+*   ✅ Has zero external dependencies (no Node.js servers, no managed DB)
+
+**Total build time:** ~4 hours  
+**Monthly cost:** $0 (free tier)  
+**Code size:** ~600 lines (Worker + Dashboard) + ~400 lines (React app)
+
+### Agent Workspace
+*   Located at `/home/user/03-projects/assistant-workspace/`
+*   Based on Cloudflare Agents SDK + TypeScript
+*   Ready for autonomous agent development
+
+---
+
+**Built with ❤️ on Cloudflare**  
+**Last updated:** August 12, 2026
@@ -0,0 +1,326 @@
+# Master Skills and Tools Documentation
+
+This document provides a comprehensive overview of all available skills, tools, and directory structures accessible in this technical workspace.
+
+## Overview
+
+This workspace is a sophisticated technical environment with extensive capabilities for development, testing, documentation, and system administration. It combines AI-powered development tools with traditional command-line interfaces and version control systems.
+
+## Knowledge Base Skills
+
+### Available Skills
+
+#### 1. **System Processing Skills**
+
+- **brainstorming-11** (Superpowers Brainstorming SKill)
+  - Purpose: Structured brainstorming and planning methodology
+  - Use case: Initial project planning and concept development
+
+- **subagent-driven-development-03** (Subagent-Driven Development)
+  - Purpose: Execute plans by dispatching fresh implementer subagents per task with review cycles
+  - Use case: Complex multi-step implementation projects requiring systematic execution
+
+- **executing-plans-01** (Executing Plans)
+  - Purpose: Load plan, review critically, execute all tasks, report completion
+  - Use case: Planning and execution of structured tasks
+
+#### 2. **Cloudflare Development Skills**
+
+- **workers-best-practices** (Workers Best Practices)
+  - Purpose: Reviews and authors Cloudflare Workers code against production best practices
+  - Use case: Cloudflare Workers development and code review
+
+- **durable-objects** (Durable Objects)
+  - Purpose: Create and review Cloudflare Durable Objects for stateful coordination
+  - Use case: Chat rooms, multiplayer games, booking systems, RPC methods
+
+- **wrangler** (Wrangler)
+  - Purpose: Cloudflare Workers CLI for deploying, developing, and managing Workers
+  - Use case: Cloudflare Workers deployment and configuration
+
+- **building-mcp-servers-on-cloudflare-21** (Building MCP Servers)
+  - Purpose: Building MCP servers on Cloudflare with updated SDK knowledge
+  - Use case: MCP server development with Cloudflare integration
+
+- **building-ai-agent-on-cloudflare** (Building AI Agent)
+  - Purpose: Building AI agents on Cloudflare with updated SDK knowledge
+  - Use case: AI agent development using Cloudflare Agents SDK
+
+#### 3. **Testing and Performance Skills**
+
+- **playwright-best-practices** (Playwright Best Practices)
+  - Purpose: Browser testing and screenshot capabilities
+  - Use case: End-to-end testing and visual regression testing
+
+- **web-perf** (Web Performance)
+  - Purpose: Analyzes web performance using Chrome DevTools
+  - Use case: Web performance auditing and optimization
+
+#### 4. **Business and User Experience Skills**
+
+- **solo-dev-fiverr** (Solo Dev Fiverr)
+  - Purpose: Power skill for solo Node/JS/TS dev selling modern dashboard and UI work
+  - Use case: Client deliverables, Fiverr gigs, premium UI development
+
+- **marketing-psychology** (Marketing Psychology)
+  - Purpose: Apply psychological principles to marketing
+  - Use case: Consumer behavior, persuasion, decision-making optimization
+
+- **ux-content-formatte-** (UX Content Formatter)
+  - Purpose: Formats text, reports, and dashboard data into scannable layouts
+  - Use case: Content formatting, report generation, data presentation
+
+#### 5. **Legal and Reference Skills**
+
+- **courtlistener-api** (Courtlistener API)
+  - Purpose: Legal case law database with PACER data and judge profiles
+  - Use case: Legal research and case law analysis
+
+#### 6. **Process Skills**
+
+- **dispatching-parallel-agents-04** (Dispatching Parallel Agents)
+  - Purpose: Delegate tasks to specialized agents with isolated context
+  - Use case: Parallel processing and specialized task execution
+
+- **find-skills** (Find Skills)
+  - Purpose: Discover and install agent skills
+  - Use case: Skill discovery and capability expansion
+
+## Available Tools
+
+### File and Directory Operations
+
+- **list_files**: Return structured listing of files and directories
+- **read_file**: Read file contents with optional line ranges and image support
+- **display_file**: Open files in viewer (for visual inspection)
+- **write_file**: Write text content to files with automatic directory creation
+- **replace_file_content**: Find and replace exact strings in files
+- **view_file**: Get file content by ID with pagination support
+- **list_processes**: List all tracked background processes
+
+### Calendar and Scheduling
+
+- **create_calendar_event**: Create calendar events, reminders, alarms
+- **search_calendar_events**: Search calendar events by text and date range
+- **update_calendar_event**: Update existing calendar events
+- **delete_calendar_event**: Delete calendar events permanently
+
+### Git and Version Control
+
+- **run_command**: Run shell commands in background with output tracking
+- **send_process_input**: Write text to process stdin
+- **kill_process**: Terminate processes
+- **get_process_status**: Get process output and status
+- **list_processes**: List all tracked background processes
+
+### Knowledge Base Management
+
+- **list_knowledge**: List knowledge bases, files, and notes
+- **query_knowledge_files**: Semantic/vector search across knowledge files
+- **view_knowledge_file**: Get content from knowledge base files
+- **search_knowledge_files**: Search files by filename across knowledge bases
+- **grep_knowledge_files**: Exact text search across knowledge files
+- **write_note**: Create new notes with markdown content
+- **view_note**: Get full note content by ID
+- **search_notes**: Search saved notes by title and content
+- **replace_note_content**: Update existing notes
+
+### Project Management
+
+- **create_tasks**: Create visible task checklists for multi-step work
+- **update_task**: Mark tasks as completed/in_progress/pending/cancelled
+
+### Documentation Generation
+
+- **writing-plans**: Create comprehensive implementation plans with task decomposition
+
+### Automation and Scheduling
+
+- **create_automation**: Create scheduled automations with iCalendar RRULE
+- **list_automations**: List scheduled automations
+- **update_automation**: Update existing automations
+- **toggle_automation**: Pause/resume automations
+- **delete_automation**: Delete automations and history
+
+### Research and Search
+
+- **search_web**: Search the public web for current information
+- **deep_research**: Multi-round research using Open WebUI web search
+- **fetch_url**: Extract main text content from web pages
+- **consult_council**: Orchestrates 3-stage council meetings
+
+### Context and Library Management
+
+- **context7_resolve-library-id**: Resolve package names to Context7-compatible library IDs
+- **context7_query-docs**: Query up-to-date documentation from Context7
+
+### Task Delegation
+
+- **delegate_task**: Delegate focused work to parallel sub-agents
+- **run_sub_agent**: Delegate tasks to sub-agents for autonomous completion
+- **run_parallel_sub_agents**: Run multiple independent sub-agent tasks concurrently
+
+### Time and Utilities
+
+- **get_current_timestamp**: Get current Unix timestamp
+- **calculate_timestamp**: Calculate timestamps for date filtering
+- **timer**: Set one-shot timers
+
+## Directory Structure
+
+### Root and Main Directories
+
+#### `/home/user/000-configs/` - System Configuration
+
+- **GH_CLI_INSTRUCTIONS.md**: GitHub CLI usage instructions
+- **PLAYWRIGHT_BROWSER_INSTRUCTIONS.md**: Playwright browser testing instructions
+- **VITE_PREVIEW_INSTRUCTIONS.md**: Vite development server preview instructions
+- **docs/**: Documentation folder with internal reference materials
+- **google/**: Directory for Google-related tools or configurations
+
+#### `/home/user/02-repos/` - Code Repositories
+
+- **shit_in_a_vault/**: Private Obsidian vault for note-taking and knowledge management
+  - Contains structured markdown documentation with YAML frontmatter
+  - Used for project documentation and knowledge retention
+
+- **webby/**: Separate web development project repository
+- **fuckgovda_vault/**: Previously deleted locally but exists remotely Obsidian vault
+
+#### `/home/user/02-repos/shit_in_a_vault/` - Obsidian Vault Structure
+
+**Vault Rules and Configuration:**
+
+- **04-models/Vault-Rules.md**: Core vault editing rules and conventions
+- **04-models/model-memory.md**: Historical context and memory
+- **000-configs/tools/**: Tool documentation and instructions
+- **01-logs/daily/**: Automated daily notes
+- **02-notes/**: Short notes, scratch work, per-topic research
+- **03-archive/**: Completed or superseded materials
+- **04-models/**: Rules, memory, boot prompt, model identity
+- **05-handoffs/**: Model-to-model transition materials
+- **06-research/**: Long-form research with citations
+- **07-tasks/**: Task tracking files with metadata
+- **00-assets/**: Attachments and media files
+
+#### `/home/user/02-repos/shit_in_a_vault/04-models/` - Model Configuration
+
+- **Vault-Rules.md**: Essential editing rules for all vault files
+- **model-memory.md**: Historical context and memory
+- Additional model configuration files may exist
+
+#### `/home/user/projects/` - Development Projects
+
+- Local project workspace for development activities
+- Temporary project directories
+- Experimentation and prototyping areas
+
+## Integration and Usage Patterns
+
+### Knowledge Management Workflow
+
+1. **Research Phase**: Use `search_web` or `deep_research` for external information
+2. **Documentation**: Store findings in `02-notes/` or `06-research/` using Obsidian vault structure
+3. **Reference**: Access knowledge via `query_knowledge_files` or `search_notes`
+4. **Automation**: Schedule documentation updates using `create_automation`
+
+### Development Workflow
+
+1. **Planning**: Use `writing-plans` for comprehensive task breakdown
+2. **Implementation**: Execute tasks via `executing-plans` or `subagent-driven-development`
+3. **Testing**: Use `playwright-best-practices` for browser testing
+4. **Version Control**: Track changes via Git commands and `run_command`
+
+### System Administration
+
+1. **Process Management**: Monitor and manage background processes with `list_processes`
+2. **Calendar Integration**: Schedule tasks and reminders with `create_calendar_event`
+3. **Documentation Maintenance**: Keep skills and tools updated using `write_file`
+
+## Capabilities Summary
+
+### Core Strengths
+
+- **Comprehensive Documentation**: Extensive tool and skill coverage
+- **Multi-modal Development**: Supports both AI-powered and traditional development
+- **Automation Integration**: Full scheduling and automation capabilities
+- **Knowledge Management**: Sophisticated Obsidian-based documentation system
+- **Testing Infrastructure**: End-to-end testing with Playwright integration
+- **Cloud Integration**: Full Cloudflare Workers and Durable Objects support
+- **Parallel Processing**: Multiple agent capabilities for concurrent development
+
+### Development Spectrum
+
+- **Simple Scripts**: Direct command-line execution
+- **Complex Projects**: Multi-agent, systematic development
+- **Documentation**: Comprehensive automated documentation generation
+- **Testing**: Full browser automation and performance analysis
+- **Deployment**: Cloud deployment with Cloudflare integration
+
+### Collaboration Features
+
+- **Task Delegation**: Multiple agent coordination
+- **Review Cycles**: Systematic review and approval processes
+- **Documentation**: Comprehensive task and project tracking
+- **Knowledge Sharing**: Repository-based knowledge management
+
+## Technical Specifications
+
+### Environment
+
+- **Operating System**: Linux 7.0.0-1010-aws (Ubuntu 22.04 base)
+- **Shell**: bash
+- **Home Directory**: `/home/user`
+- **Primary Development Areas**: `/home/user`, `/home/user/projects`, `/home/user/repos`
+
+### Authentication and Access
+
+- **SSH Key**: Available for secure connections
+- **GitHub CLI**: v2.96.0 authenticated as `stateofshit`
+- **Access Levels**: Multiple access tiers for different environments
+
+### Available Technologies
+
+- **Languages**: Node.js, Python 3.12.13, bash
+- **Frameworks**: Vite, React (preview available at `https://preview.boogerclub.com`)
+- **Testing**: Playwright browser automation
+- **Deployment**: Cloudflare Workers integration
+- **Documentation**: Obsidian vault with markdown and YAML support
+
+### Preview and Testing
+
+- **Development Server**: Vite development server with preview at `https://preview.boogerclub.com`
+- **Browser Testing**: Playwright screenshot and full-page testing capabilities
+- **Performance Analysis**: Chrome DevTools integration for web performance
+
+## Usage Recommendations
+
+### For Beginners
+
+1. Start with simple tasks using `run_command` for basic operations
+2. Explore documentation in `/home/user/000-configs/`
+3. Use `list_files` to understand directory structure
+4. Begin with `write_note` for simple note-taking
+
+### For Intermediate Users
+
+1. Explore `subagent-driven-development-03` for complex projects
+2. Use `search_web` for external research
+3. Implement automation with `create_automation`
+4. Develop comprehensive documentation with `writing-plans`
+
+### For Advanced Users
+
+1. Utilize parallel processing with `run_parallel_sub_agents`
+2. Implement sophisticated workflows with `delegate_task`
+3. Create complex Cloudflare Workers with `wrangler`
+4. Build advanced automation systems with `durable-objects`
+
+## Contact and Support
+
+For assistance with this system:
+
+- Refer to specific tool documentation in `/home/user/000-configs/`
+- Use `search_notes` to find relevant documentation
+- Consult `context7_query-docs` for library-specific information
+- Contact through GitHub repositories for code-specific issu
@@ -0,0 +1,207 @@
+---
+title: "kycourt-colab — Kentucky Appellate Court Corpus Pipeline (project notes)"
+status: "active"
+folder: "repo root (kycourt-colab)"
+tags: [kycourt, pipeline, colab, project-notes]
+created: "2026-08-12"
+updated: "2026-08-12"
+version: "1.1.0"
+---
+
+# kycourt-colab — Kentucky Appellate Court Corpus Pipeline
+
+## Overview
+
+A single **generated Google Colab notebook** (`ky_court_pipeline.ipynb`) that
+restores, discovers, downloads, extracts, and verifies the full Kentucky
+appellate court document corpus (~**98.7k documents**). It is built to survive
+Colab's ephemeral runtimes: it auto-resumes after disconnects, is idempotent,
+and mirrors all state to Google Drive so nothing is lost between sessions.
+
+**Remote:** https://github.com/stateofshit/kycourt-colab (private)
+**Local worktree:** `/home/user/00-incoming/kycourt-colab`
+
+## What problem it solves
+
+A batch archival crawl of public court documents over a ~10-hour-per-session
+horizon, on infrastructure (Colab) that kills your process mid-run. Requirements
+that fall out of that:
+
+- Every session must pick up exactly where the last one stopped.
+- Re-running must be safe (idempotent) — no double-downloads, no data loss.
+- State must live somewhere durable (Drive), because the runtime is throwaway.
+- The crawl must be polite to a public API (paced, backoff, retries) and
+  deduplicate aggressively (content-addressed storage).
+
+## Architecture
+
+The notebook is **not hand-written** — its cells are inlined from real Python
+modules in `notebook_builder/cells/` (one module per cell, `c00` → `c90`), and
+regenerated with `notebook_builder/build.py`. This means the pipeline is
+version-controlled, unit-testable Python that *just happens to be shipped as* a
+Colab notebook.
+
+Two shared abstractions carry every stage:
+
+- **SQLite DB** (`ky_court.sqlite`) — source of truth: `documents`,
+  `discovery_runs`, `extraction_log`, `kv`. Opened in **WAL mode** with a
+  30s `busy_timeout` so a background reader can snapshot it while the writer
+  commits (see concurrency below).
+- **`Store` abstraction** — Google Drive in Colab, local filesystem in tests;
+  exposes `exists / read_file / write_bytes / upload_file / list_dir`. All
+  stages talk to it, never to Drive directly.
+
+**Stage order:** `restore → discover → backfill → download → extract → sync/verify`.
+
+## The six stages (what each actually does)
+
+### 1. Restore — `c40_restore.py`
+Rehydrate a fresh/empty workspace from the newest `ky_court-WIP-resume-*.tar.gz`.
+Only acts when no DB exists; skips otherwise. If a stale workspace dir is
+present it is **moved aside to `-pre-restore` (renamed, never deleted)** before
+extracting — the "never destroys data" guarantee. Extracts to temp, then moves
+into place.
+
+### 2. Discover — `c50_discover.py`
+Enumerate the corpus by searching the court API so we know *what exists*.
+Builds search shards (`YYYY-CA`, `YYYY-SC` per year; ceiling-hit shards get
+sub-sharded `YYYY-CA-0..9`). Pages results with `X-CTrack-Paging-*` headers up
+to a 10k ceiling; a `400` or ceiling-cross raises to force sub-sharding rather
+than silently missing docs. Normalizes rows into `documents` with
+`status='discovered'`, inserted idempotently via `ON CONFLICT(document_id)`.
+Sweeps repeatedly until a pass adds fewer than `CONVERGENCE_DELTA` docs
+(convergence). Every sweep logged to `discovery_runs`.
+
+### 3. Backfill — `c60_backfill.py`
+Deduplicate *existing* backups into the content-addressed store. Hashes every
+file in `pdf_backup/` (SHA-256), writes unique content to `pdf_by_hash/<sha>.pdf`
+(9,582 backup files → 3,326 unique PDFs) — "do we already have this?" becomes a
+hash lookup. Normalizes DB `local_pdf` pointers. Guarded by `kv backfill_done`.
+
+### 4. Download — `c70_download.py`
+Fetch PDFs for all `discovered`/`failed` docs with `attempts < MAX_ATTEMPTS`,
+in batches of 50. `api_download` does a `HEAD` (rejects non-200 and anything
+> `MAX_PDF_BYTES`), then streams `GET`, hashing as it goes. Dedup wins: a hash
+already in the store/manifest uploads nothing. **Retry/backoff** on transient
+errors: `sleep = min(60, 5·2^attempts)` (10→20→40→60s, capped), up to 4
+attempts. Permanent errors (400/401/403/404/**413**) fail immediately, no
+backoff. Writes to `tmp/<id>.part`, promotes to store, paces itself, and pushes
+a heartbeat to `status.json` every 100 attempts.
+
+### 5. Extract — `c80_extract.py`
+Turn downloaded PDFs into searchable text with `pypdf`. Batches of 500 with 8
+parallel worker threads. A PDF with no text layer is a valid `"empty"` result,
+not an error. Text written to `text/<sha>.txt`; `extracted_chars` recorded.
+Heuristic flags likely-scanned (image-only) PDFs when `chars/pages < 20`.
+
+### 6. Sync/Verify — `c90_verify.py`
+Prove nothing is corrupted and push an authoritative snapshot + summaries to
+Drive. Runs `PRAGMA integrity_check`; counts totals and the **remaining
+queues** (download + extract eligibility queries mirror the stage code exactly,
+so `done` is trustworthy). Exports `document_inventory.csv` +
+`failed_inventory.csv`. `checkpoint()` makes a consistent DB snapshot via
+`src.backup(dst)` (SQLite online backup API) and uploads it as the new
+`ky_court.sqlite`. Updates `status.json` with `done = (queues empty)`.
+Prints `PASS`/`FAIL`.
+
+## Concurrency (the interesting part)
+
+`checkpoint()` uses SQLite's **online backup API** (`src.backup(dst)`) from a
+**separate connection** — not `copytree`, not a WAL checkpoint. It copies the
+DB page-by-page and re-copies any page the writer modified during the pass,
+looping until a clean pass completes. Combined with **WAL mode** (readers don't
+block the writer) and **`busy_timeout=30000`**, this yields a
+transactionally-consistent snapshot even while Download is committing every doc.
+A naive file copy would produce a torn/corrupt snapshot and drop WAL-committed
+data.
+
+## Key design decisions
+
+- **Idempotent + resume-safe everywhere**: skip what's done, never auto-delete
+  on Drive.
+- **Named permanent-failure set** (`PERMANENT_STATUSES` + `_is_permanent()`)
+  distinguishes never-succeed (401/403/404/413) from transient (5xx/conn/429/408).
+- **Content-addressed store** (`pdf_by_hash/<sha>.pdf`) makes dedup a lookup.
+- **Config via one `CFG` dict** with env overrides (`KY_BUDGET`, `KY_DRY`,
+  `KY_PHASES`, `KY_RETRY_HARD`, `KY_LIVE`).
+
+## Config knobs (top cell `CFG`)
+
+| Key | Default | Meaning |
+|---|---|---|
+| `BUDGET_SECONDS` | 10h | max wall-clock per session (`KY_BUDGET`) |
+| `SHUTDOWN_SECONDS` | 15 min | stop early so the session finishes cleanly |
+| `PACING_SEARCH` / `PACING_DL` | (0.3,0.7)/(0.3,0.5) | random sleep between API calls |
+| `MAX_ATTEMPTS` | 4 | download retries before `failed` |
+| `MAX_PDF_BYTES` | 200 MiB | size cap; oversized (413) fails permanently, no retry/backoff |
+| `MAX_EXTRACT_ATTEMPTS` | 3 | extraction retries before skip |
+| `RETRY_HARD` | 0 (`KY_RETRY_HARD`) | on resume, reset `failed` → `discovered` |
+| `PHASES` | discover,backfill,download,extract | stages to run (`KY_PHASES`) |
+| `DRY_RUN` | 0 (`KY_DRY`) | 1 = fully offline, no network |
+
+## File layout
+
+```
+ky_court_pipeline.ipynb         the deliverable (generated)
+notebook_builder/
+  build.py                       regenerate the notebook from cells
+  cells/  c00_config … c90_verify + drive_store.py
+tests/  test_pipeline_local.py + conftest.py (offline; live gated by KY_LIVE)
+scripts/  smoke_local.py         full offline pipeline over the real DB
+RUNBOOK.md                      operational runbook (first run, resuming, failure modes)
+docs/superpowers/ specs/ + plans/  design spec + implementation plan
+PROJECT_NOTES.md                 this file
+.gitignore                       ignores .venv, extracted/, *.sqlite, __pycache__,
+                                 .worktrees/, .superpowers/  (NOT the WIP tars)
+```
+
+> **Data is NOT in the repo.** `extracted/`, `*.tar.gz` resume/WIP archives, and
+> `sanitize-work/` live only in Drive / locally. Gitignore does not cover the
+> WIP `.tar.gz` files, so never `git add -A` from the repo root.
+
+## Running / testing
+
+```bash
+python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
+.venv/bin/pytest tests -q                # 20 offline tests (live ones skipped)
+.venv/bin/python scripts/smoke_local.py  # full pipeline over the real 98,724-row DB
+.venv/bin/python notebook_builder/build.py  # regenerate the notebook
+```
+
+Live API tests are gated behind `KY_LIVE=1` and share a ≤10-request budget.
+Suite result at v1.2: **20 passed, 3 skipped**.
+
+## Failure modes (from RUNBOOK)
+
+- **Colab disconnect mid-run** → re-run; `recover()` heals stale rows, drops
+  `.part` files.
+- **Drive quota exceeded** → checkpoint push fails; run continues, store lags.
+- **5xx/connection storms** → backoff `min(60, 5·2^attempts)` up to `MAX_ATTEMPTS`.
+- **Search shard hits 10k ceiling** → planner sub-shards next run.
+- **`integrity_check` not "ok"** → DB corrupt; restore from last resume tar.
+
+## Versioning / releases
+
+Immutable milestone tags (never force-move a published tag):
+
+```
+pipeline-v1  33a6ec1  original pipeline milestone
+v1.1         ae77344  +README
+v1.2         ec248fe  +413 oversize permanent-failure fix (MAX_PDF_BYTES knob)
+```
+
+`v1.2` also has a **GitHub release** with changelog:
+https://github.com/stateofshit/kycourt-colab/releases/tag/v1.2
+
+## Status
+
+Project complete and shipped. Suite green offline. The pipeline is ready to run in Colab (restore from the resume tar → Run all).
+
+### Roadmap / Future enhancements
+- **Parallel extract workers** — tune the 8-worker batch size based on observed Colab runtime limits.
+- **Drive quota monitoring** — automated alert when `status.json` reports approaching quota, with graceful degradation to skip upload.
+- **CLI entry point** — `kycourt-pipeline run --phase discover` to run individual stages outside Colab.
+- **Query layer** — lightweight read-only API over the SQLite DB for inspecting discovered/extracted docs without re-running the pipeline.
+- **Vault migration** — if/when the repo is folded into the vault, adapt `folder` frontmatter and consolidate `.tar.gz` WIP archives under the vault's resume format. Remote backup, README, tags, and v1.2 release all
+in place on GitHub. Suite green offline. The pipeline is ready to run in Colab
+(restore from the resume tar → Run all).
@@ -0,0 +1,59 @@
+# ky-court-colab
+
+A single generated Google Colab notebook that **restores, discovers, downloads,
+extracts, and verifies** the full Kentucky appellate court document corpus
+(~98.7k documents). Built to survive Colab disconnects — it auto-resumes, is
+idempotent, and mirrors all state to Google Drive so nothing is lost between
+sessions.
+
+## What it does
+
+`ky_court_pipeline.ipynb` walks the corpus through six stages:
+
+```
+restore → discover → backfill → download → extract → sync/verify
+```
+
+Every stage reads/writes one SQLite DB and a `Store` abstraction (Google Drive
+in Colab, local filesystem in tests). Any stage stops gracefully at the session
+time budget; re-running resumes exactly where it left off.
+
+## Highlights
+
+- **Resumable** — `recover()` heals stale state; each stage skips what's done.
+- **Idempotent** — safe to re-run as often as you like.
+- **Drive-backed** — SQLite DB, deduped PDFs, extracted text, exports, and a
+  live `status.json` heartbeat all live under `MyDrive/ky_court-WIP/`.
+- **Never auto-deletes on Drive** — original backups are retained.
+- **Polite + fault-tolerant** — paced API calls, exponential backoff, capped
+  retries, and a `DRY_RUN` mode for fully offline exercise.
+
+## Layout
+
+| Path | Purpose |
+|---|---|
+| `ky_court_pipeline.ipynb` | the deliverable (generated notebook) |
+| `notebook_builder/` | sources: `build.py` + one module per cell (`c00_config` … `c90_verify`) |
+| `tests/` | offline pytest suite (18 tests; live ones gated behind `KY_LIVE=1`) |
+| `scripts/smoke_local.py` | full offline pipeline over the real DB |
+| `RUNBOOK.md` | full operational runbook (first run, resuming, failure modes, config knobs) |
+
+## Quick start
+
+```bash
+# local test/smoke
+python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
+.venv/bin/pytest tests -q
+.venv/bin/python scripts/smoke_local.py
+.venv/bin/python notebook_builder/build.py   # regenerate the notebook
+
+# on Google Colab
+# upload ky_court-WIP-resume-*.tar.gz to MyDrive/ (if Drive is empty),
+# open ky_court_pipeline.ipynb, Runtime > Run all.
+```
+
+See **`RUNBOOK.md`** for the full runbook — first run, resuming, what "done"
+looks like, failure modes, and every config knob.
+
+> **Note:** input data (extracted archives, resume tarballs, `_WIP` backups)
+> is deliberately **not** in this repo — it lives in Google Drive / locally.
@@ -0,0 +1,116 @@
+# RUNBOOK — ky_court_pipeline.ipynb
+
+One notebook to restore, discover, download, extract, sync, and verify the full
+Kentucky appellate court document corpus (~98.7k documents). It is designed to
+run in Google Colab, auto-resume after disconnects, and mirror all state up to a
+Drive folder so nothing is lost across sessions.
+
+## What this is
+
+A single generated notebook (`ky_court_pipeline.ipynb`) whose cells are inlined
+from real Python modules in `notebook_builder/cells/`. The notebook is rebuilt
+with `notebook_builder/build.py`. All stages read/write the same SQLite DB and a
+`Store` abstraction that is Google Drive in Colab and the local filesystem in
+tests/smoke.
+
+Stage order: **restore → discover → backfill → download → extract → sync/verify**.
+Any stage stops gracefully at the session budget; re-running resumes where it
+left off.
+
+## Files on Drive (`MyDrive/ky_court-WIP/`)
+
+| Path | Purpose |
+|---|---|
+| `ky_court.sqlite` | source of truth (documents, discovery_runs, kv, extraction_log) |
+| `pdf_by_hash/<sha256>.pdf` | deduped PDFs keyed by content hash (3,326 unique from 9,582 backup files) |
+| `pdf_backup/<case>_<n>.pdf` | original named backups — **retained, never deleted** |
+| `text/<sha256>.txt` | extracted text per PDF |
+| `exports/file_manifest.csv` | sha256 → size manifest |
+| `exports/document_inventory.csv` / `failed_inventory.csv` | exported summaries from verify |
+| `status.json` | live heartbeat + phase/counts (the "is it alive" check) |
+| `ky_court-WIP-resume-*.tar.gz` | (MyDrive root) resume tarball used to rehydrate a fresh runtime |
+
+The store root is `MyDrive/ky_court-WIP` (Colab) or `<WORKDIR>/ky_court-WIP` (local).
+
+## First run
+
+1. If Drive is empty (no `ky_court-WIP`), upload `ky_court-WIP-resume-YYYY-MM-DD.tar.gz`
+   to `MyDrive/`. Stage **restore** will extract it.
+2. Open `ky_court_pipeline.ipynb` in Colab.
+3. Edit the first code cell (`CFG`) if needed (budget, phases — see Config knobs).
+4. **Runtime > Run all**. A session mounts Drive, opens the DB (pulling it from
+   the store if absent), heals stale state, then runs the enabled phases.
+
+Expected session timeline (typical):
+- setup: ~1 min (Drive mount, dirs, recover)
+- discover: 10–30 min (74 shards, paged search, paced)
+- backfill: ~20 min (hash `pdf_backup` → dedupe into `pdf_by_hash`)
+- download: ~10 h per session ≈ 6–12k documents (budget-capped)
+- extract: rides along between batches (thread-pooled PDF text extraction)
+- verify: ~5 min (integrity check, inventory exports, status push)
+
+## Resuming
+
+Just re-run the notebook. `recover()` closes any stale `running` discovery runs,
+drops leftover `.part` files, and each stage skips what's already done:
+- backfill: guarded by `kv backfill_done`
+- download: only `discovered`/`failed` docs with `attempts < MAX_ATTEMPTS`
+- extract: only `downloaded` docs with no extracted text yet
+
+Idempotent by design — safe to rerun as often as you like.
+
+## What "done" looks like
+
+Verify prints `PASS {checks}` and, when every queue is empty:
+- notebook verify output includes `done: True`
+- `status.json` on Drive contains `"done": true`
+
+A session that merely hit its budget is **not** "done" — it stops early with
+`status.json` showing the phase and remaining queues. That is normal mid-flight
+state, not an error.
+
+## Failure modes
+
+- **Colab disconnect mid-run** → just re-run. `recover()` heals stale rows;
+  `.part` files are dropped; partial downloads retry from the queue.
+- **Drive quota exceeded** → verify/checkpoint push fails; the run continues
+  locally in the session but the store lags. Check Drive quota, free space, re-run.
+- **5xx / connection storms** → `api_download` returns an error tuple and
+  `stage_download` backs off `min(60, 5*2^attempts)` seconds, retrying up to
+  `MAX_ATTEMPTS` before marking the doc `failed`.
+- **Search shard hits the 10k ceiling** → `api_search_pages` raises; the planner
+  sub-shards that seed (`YYYY-CA-N`) on the next run.
+- **`integrity` check not "ok"** → verify prints `FAIL`; the DB is corrupted and
+  should be restored from the last resume tar before trusting anything.
+
+## Config knobs (top code cell `CFG`)
+
+| Key | Default | Meaning |
+|---|---|---|
+| `BUDGET_SECONDS` | 10 * 3600 | max wall-clock per session (env `KY_BUDGET`) |
+| `SHUTDOWN_SECONDS` | 900 | stop stages this long before the deadline so the session can finish cleanly |
+| `PACING_SEARCH` / `PACING_DL` | (0.3,0.7) / (0.3,0.5) | random sleep range between API calls (polite throttling) |
+| `MAX_ATTEMPTS` | 4 | download retries before `failed` |
+| `MAX_PDF_BYTES` | 200 MiB | size cap; oversized (HTTP 413) fails permanently, no retry/backoff |
+| `MAX_EXTRACT_ATTEMPTS` | 3 | PDF-extraction retries before skip |
+| `RETRY_HARD` | 0 (env `KY_RETRY_HARD`) | on resume, reset `failed` docs back to `discovered` |
+| `PHASES` | discover,backfill,download,extract | comma-separated phases to run (env `KY_PHASES`) |
+| `DRY_RUN` | 0 (env `KY_DRY`) | 1 = fully offline: discovery/download skip all network |
+
+## Safety notes
+
+- The pipeline **never deletes on Drive automatically**. `pdf_backup` is retained;
+  pre-restore move-aside only renames (`-pre-restore`), never deletes.
+- Set `DRY_RUN=1` (`KY_DRY=1`) to exercise the whole flow offline with zero
+  network calls — the smoke script (`scripts/smoke_local.py`) does exactly this.
+- Live API tests are gated behind `KY_LIVE=1` and share a ≤10-request budget;
+  keep them there so the ordinary test suite stays offline and fast.
+
+## Local / tests
+
+```bash
+python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
+.venv/bin/pytest tests -q               # 18 offline tests (live ones skipped)
+.venv/bin/python scripts/smoke_local.py # full pipeline over the real 98,724-row DB
+.venv/bin/python notebook_builder/build.py  # regenerate the notebook
+```
@@ -0,0 +1,40 @@
+---
+created: 2026-08-06T14:38:00
+tags:
+  - hosts
+  - WSL
+  - windows
+updated: 2026-08-06T14:44:00
+---
+
+# WSL 
+user=medic
+ Host openwebui
+	HostName 18.223.53.133
+	User ubuntu
+	IdentityFile ~/.ssh/openwebui_wsl
+Host github-debtcoder
+	HostName github.com
+	User git
+	IdentityFile ~/.ssh/id_ed25519_github
+Host github-plug-ooo
+	Hostname github.com
+	User git
+	IdentityFile ~/.ssh/openwebui_wsl
+
+
+
+-----------------------
+# Windows
+user:there
+Host dashit
+    HostName 3.23.239.121
+    User ubuntu
+    IdentityFile ~/.ssh/dashit.pem
+    ServerAliveInterval 60
+# WSL
+Host dashit
+    HostName 3.23.239.121
+    User ubuntu
+    IdentityFile ~/.ssh/dashit.pem
+    ServerAliveInterval 60
@@ -1,3 +1,7 @@
+---
+created: 2026-08-06T15:28:00
+updated: 2026-08-12T15:28:00
+---

 ---

@@ -160,4 +164,20 @@ DRIVE_API_KEY=AIzaSyAB5mry4SGDb1bmNGLGZ4bmDPXAUai6HPs
 # medic8dcloud@gmail.com
 ## pinecone
 KEY=pcsk_Ats6a_7tukrjtHVJskhZDG8zVDBjG5X4PNPrVECaGVRU3B137x432U62SfbV2495Psknd
+```
+```
+# thestateofshit@gmail.com
+render_api_key=rnd_PyfjhLC3KKG43qazh5TbA9aJsO62
+
+
+
+# medic8dcloud@gmail.com google ai studio
+key=AIzaSyDZfMTQ0ayi3phYTKwdpI7uPPtDYfdvgZk
+key=8RN6IOh97RbGeoixINNWp7QRtt-_LBKmJD-xuhugZ1XurDZA
+## for drive,docs,sheets,gmail,calander
+api_key=AIzaSyASs9IQteSVtV61fkUAR9mu9JkB0sMw1Vw
+---
+# medic8cloud qdrant
+apikey=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3MiOiJtIiwic3ViamVjdCI6ImFwaS1rZXk6NjgzYTc4ZGQtNmY4Mi00ZWMxLTg0ODYtNWU3YmY0NGY0OGRhIn0.WiO_I07EuUSDQVI1w2VzxBkXTPyIEUfkWtc_DU_gmQk
+enpoint=https://0ff33b75-b828-417f-8c60-e219d442ee2d.us-east-1-1.aws.cloud.qdrant.io
 ```
@@ -0,0 +1,106 @@
+# oracle vps
+	commands
+	free -h
+	df -h
+	sudo ss -tulpn
+## before docker and vaultwarden
+```
+               total        used        free      shared  buff/cache   available
+Mem:           954Mi       401Mi       184Mi       5.0Mi       532Mi       552Mi
+Swap:          2.0Gi       2.0Mi       2.0Gi
+Filesystem      Size  Used Avail Use% Mounted on
+tmpfs            96M  1.1M   95M   2% /run
+efivarfs        256K   21K  231K   9% /sys/firmware/efi/efivars
+/dev/sda1        45G  4.9G   40G  12% /
+tmpfs           478M     0  478M   0% /dev/shm
+tmpfs           5.0M     0  5.0M   0% /run/lock
+/dev/sda16      881M  156M  664M  19% /boot
+/dev/sda15      105M  6.2M   99M   6% /boot/efi
+tmpfs            96M   12K   96M   1% /run/user/1002
+tmpfs            96M   12K   96M   1% /run/user/1001
+Netid         State          Recv-Q         Send-Q                    Local Address:Port                 Peer Address:Port        Process                                                                                                                           
+udp           UNCONN         0              0                               0.0.0.0:111                       0.0.0.0:*            users:(("rpcbind",pid=696,fd=5),("systemd",pid=1,fd=238))                                                                        
+udp           UNCONN         0              0                            127.0.0.54:53                        0.0.0.0:*            users:(("systemd-resolve",pid=712,fd=16))                                                                                        
+udp           UNCONN         0              0                         127.0.0.53%lo:53                        0.0.0.0:*            users:(("systemd-resolve",pid=712,fd=14))                                                                                        
+udp           UNCONN         0              0                      10.0.13.200%ens3:68                        0.0.0.0:*            users:(("systemd-network",pid=805,fd=21))                                                                                        
+udp           UNCONN         0              0                                  [::]:111                          [::]:*            users:(("rpcbind",pid=696,fd=7),("systemd",pid=1,fd=241))                                                                        
+tcp           LISTEN         0              4096                      127.0.0.53%lo:53                        0.0.0.0:*            users:(("systemd-resolve",pid=712,fd=15))                                                                                        
+tcp           LISTEN         0              4096                         127.0.0.54:53                        0.0.0.0:*            users:(("systemd-resolve",pid=712,fd=17))                                                                                        
+tcp           LISTEN         0              4096                            0.0.0.0:22                        0.0.0.0:*            users:(("sshd",pid=1090,fd=3),("systemd",pid=1,fd=228))                                                                          
+tcp           LISTEN         0              4096                            0.0.0.0:111                       0.0.0.0:*            users:(("rpcbind",pid=696,fd=4),("systemd",pid=1,fd=237))                                                                        
+tcp           LISTEN         0              511                             0.0.0.0:80                        0.0.0.0:*            users:(("nginx",pid=7514,fd=5),("nginx",pid=7513,fd=5),("nginx",pid=7510,fd=5))                                                  
+tcp           LISTEN         0              4096                               [::]:22                           [::]:*            users:(("sshd",pid=1090,fd=4),("systemd",pid=1,fd=229))                                                                          
+tcp           LISTEN         0              4096                               [::]:111                          [::]:*            users:(("rpcbind",pid=696,fd=6),("systemd",pid=1,fd=239))                                                                        
+tcp           LISTEN         0              511                                [::]:80                           [::]:*            users:(("nginx",pid=7514,fd=6),("nginx",pid=7513,fd=6),("nginx",pid=7510,fd=6))                                                  
+```
+## after install
+```
+             total        used        free      shared  buff/cache   available
+Mem:           954Mi       443Mi        62Mi       4.7Mi       612Mi       510Mi
+Swap:          2.0Gi        43Mi       2.0Gi
+Filesystem      Size  Used Avail Use% Mounted on
+tmpfs            96M  1.2M   95M   2% /run
+efivarfs        256K   21K  231K   9% /sys/firmware/efi/efivars
+/dev/sda1        45G  5.6G   39G  13% /
+tmpfs           478M     0  478M   0% /dev/shm
+tmpfs           5.0M     0  5.0M   0% /run/lock
+/dev/sda16      881M  156M  664M  19% /boot
+/dev/sda15      105M  6.2M   99M   6% /boot/efi
+tmpfs            96M   12K   96M   1% /run/user/1002
+tmpfs            96M   12K   96M   1% /run/user/1001
+Netid         State          Recv-Q         Send-Q                    Local Address:Port                 Peer Address:Port        Process                                                                                                                           
+udp           UNCONN         0              0                               0.0.0.0:111                       0.0.0.0:*            users:(("rpcbind",pid=696,fd=5),("systemd",pid=1,fd=222))                                                                        
+udp           UNCONN         0              0                            127.0.0.54:53                        0.0.0.0:*            users:(("systemd-resolve",pid=712,fd=16))                                                                                        
+udp           UNCONN         0              0                         127.0.0.53%lo:53                        0.0.0.0:*            users:(("systemd-resolve",pid=712,fd=14))                                                                                        
+udp           UNCONN         0              0                      10.0.13.200%ens3:68                        0.0.0.0:*            users:(("systemd-network",pid=805,fd=21))                                                                                        
+udp           UNCONN         0              0                                  [::]:111                          [::]:*            users:(("rpcbind",pid=696,fd=7),("systemd",pid=1,fd=229))                                                                        
+tcp           LISTEN         0              511                             0.0.0.0:443                       0.0.0.0:*            users:(("nginx",pid=10572,fd=10),("nginx",pid=10571,fd=10),("nginx",pid=10540,fd=10))                                            
+tcp           LISTEN         0              4096                      127.0.0.53%lo:53                        0.0.0.0:*            users:(("systemd-resolve",pid=712,fd=15))                                                                                        
+tcp           LISTEN         0              4096                         127.0.0.54:53                        0.0.0.0:*            users:(("systemd-resolve",pid=712,fd=17))                                                                                        
+tcp           LISTEN         0              4096                          127.0.0.1:8000                      0.0.0.0:*            users:(("docker-proxy",pid=10813,fd=8))                                                                                          
+tcp           LISTEN         0              4096                            0.0.0.0:22                        0.0.0.0:*            users:(("sshd",pid=1090,fd=3),("systemd",pid=1,fd=193))                                                                          
+tcp           LISTEN         0              4096                            0.0.0.0:111                       0.0.0.0:*            users:(("rpcbind",pid=696,fd=4),("systemd",pid=1,fd=221))                                                                        
+tcp           LISTEN         0              511                             0.0.0.0:80                        0.0.0.0:*            users:(("nginx",pid=10572,fd=5),("nginx",pid=10571,fd=5),("nginx",pid=10540,fd=5))                                               
+tcp           LISTEN         0              511                                [::]:443                          [::]:*            users:(("nginx",pid=10572,fd=9),("nginx",pid=10571,fd=9),("nginx",pid=10540,fd=9))                                               
+tcp           LISTEN         0              4096                               [::]:22                           [::]:*            users:(("sshd",pid=1090,fd=4),("systemd",pid=1,fd=194))                                                                          
+tcp           LISTEN         0              4096                               [::]:111                          [::]:*            users:(("rpcbind",pid=696,fd=6),("systemd",pid=1,fd=225))                                                                        
+tcp           LISTEN         0              511                                [::]:80                           [::]:*            users:(("nginx",pid=10572,fd=6),("nginx",pid=10571,fd=6),("nginx",pid=10540,fd=6))                                               
+```
+
+# after Jackett
+```
+Netid      State       Recv-Q      Send-Q              Local Address:Port           Peer Address:Port      Process
+udp        UNCONN      0           0                         0.0.0.0:111                 0.0.0.0:*
+udp        UNCONN      0           0                      127.0.0.54:53                  0.0.0.0:*
+udp        UNCONN      0           0                   127.0.0.53%lo:53                  0.0.0.0:*
+udp        UNCONN      0           0                10.0.13.200%ens3:68                  0.0.0.0:*
+udp        UNCONN      0           0                            [::]:111                    [::]:*
+tcp        LISTEN      0           511                       0.0.0.0:443                 0.0.0.0:*
+tcp        LISTEN      0           4096                      0.0.0.0:9117                0.0.0.0:*
+tcp        LISTEN      0           4096                127.0.0.53%lo:53                  0.0.0.0:*
+tcp        LISTEN      0           4096                   127.0.0.54:53                  0.0.0.0:*
+tcp        LISTEN      0           4096                    127.0.0.1:8000                0.0.0.0:*
+tcp        LISTEN      0           4096                      0.0.0.0:22                  0.0.0.0:*
+tcp        LISTEN      0           4096                      0.0.0.0:111                 0.0.0.0:*
+tcp        LISTEN      0           511                       0.0.0.0:80                  0.0.0.0:*
+tcp        LISTEN      0           511                          [::]:443                    [::]:*
+tcp        LISTEN      0           4096                         [::]:9117                   [::]:*
+tcp        LISTEN      0           4096                         [::]:22                     [::]:*
+tcp        LISTEN      0           4096                         [::]:111                    [::]:*
+tcp        LISTEN      0           511                          [::]:80                     [::]:*
+git@instance-20260807-0848:~$   df -h
+Filesystem      Size  Used Avail Use% Mounted on
+tmpfs            96M  1.3M   95M   2% /run
+efivarfs        256K   21K  231K   9% /sys/firmware/efi/efivars
+/dev/sda1        45G  6.1G   39G  14% /
+tmpfs           478M     0  478M   0% /dev/shm
+tmpfs           5.0M     0  5.0M   0% /run/lock
+/dev/sda16      881M  156M  664M  19% /boot
+/dev/sda15      105M  6.2M   99M   6% /boot/efi
+tmpfs            96M   12K   96M   1% /run/user/1001
+tmpfs            96M   12K   96M   1% /run/user/1002
+git@instance-20260807-0848:~$   free -h
+               total        used        free      shared  buff/cache   available
+Mem:           954Mi       518Mi        71Mi        30Mi       556Mi       435Mi
+Swap:          2.0Gi       234Mi       1.8Gi
+```