vault backup: 2026-08-12 15:38:55
This commit is contained in:
@@ -1,145 +0,0 @@
|
|||||||
# Kentucky Unemployment Insurance: Comprehensive Catalog of Agency Ministerial Duties
|
|
||||||
|
|
||||||
This catalog identifies and compiles every mandatory, non-discretionary action and obligation (**"shall"** or **"must"** requirements) that the Kentucky Office of Unemployment Insurance (OUI) and the Kentucky Unemployment Insurance Commission (KUIC) are legally required to perform under Kentucky Revised Statutes (KRS Chapter 341) and Kentucky Administrative Regulations (787 KAR Chapters 1, 2, and 3).
|
|
||||||
|
|
||||||
To preserve substantial rights, enforce accountability, and ensure due process, the agency's operations are divided into six logical domains.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. Claim Intake, Processing, & Claimant Advisory
|
|
||||||
|
|
||||||
These duties govern the initial intake of claims, mandatory eligibility screenings, required notifications to affected employers, and required counseling for displaced workers.
|
|
||||||
|
|
||||||
| # | Duty | Source & Requirement | Notes & Operational Context |
|
|
||||||
|---|---|---|---|
|
|
||||||
| **1** | **Immediate Separation Notification to Employers** | **787 KAR 1:060 Section 1(1):** "If an initial claim for benefits is filed by a claimant or if a reopened claim for benefits is filed... the Office of Unemployment Insurance **shall** immediately notify the claimant's most recent employer in writing of the filing." [21] | **Separation for Cause Reports:** This immediate notification triggers the employer's clock to submit protests regarding misconduct or quits [21, 24]. |
|
|
||||||
| **2** | **Secondary Separations Notice (Ten-Week Rule)** | **787 KAR 1:060 Section 1(2):** "If the claimant has worked for his or her most recent employer for less than ten (10) weeks, the office **shall** also notify his or her next most recent employer in writing of the claim filing." [21] | **Liability Allocation:** Ensures that any previous employer who may be charged for benefits is notified of the potential claim [21, 447]. |
|
|
||||||
| **3** | **Claim Backdating for Good Cause** | **787 KAR 1:090 Section 2(4):** "...upon the presentation by the unemployed worker of reasons found to constitute good cause for failure to file at an earlier date, the secretary **shall** backdate the initial or reopened claim to the first day of the week in which the worker became unemployed, or the second calendar week preceding the date the worker filed, whichever is later." [37] | ** claimant Relief:** Good cause explicitly includes illness, personal availability barriers, or internet/phone system outages [37]. |
|
|
||||||
| **4** | **Mandatory Direction to File by Mail** | **787 KAR 1:090 Section 3(3)(a):** "The secretary **shall** direct an unemployed worker to claim benefits by mail if it is not possible for the worker to claim by either [internet or telephone portals] due to: 1. Unavailability of those options... 2. Unavailability... due to technical problems; or 3. A physical or mental condition preventing the worker from using those options." [44] | **Accessibility Protection:** Mandates a paper-based backup method to ensure claimants with disabilities or technical hurdles are not locked out [44]. |
|
|
||||||
| **5** | **Notice of Late-Filing Allowances** | **787 KAR 1:090 Section 6(1):** "The secretary **shall** notify an unemployed worker if the secretary determines that the unemployed worker failed to file a claim... within the specified time due to: (a) The employer's failure to comply... (b) Coercion or intimidation... (c) Failure by the Office... personnel to discharge necessary responsibilities." [47] | **Agency/Employer Fault:** Provides a 14-day grace period to file a retroactive claim if the delay was caused by third-party interference or agency error [48]. |
|
|
||||||
| **6** | **Mandatory Advices of Educational & Training Resources** | **KRS 341.102:** "At the time of being notified of a valid claim, the Office of Unemployment Insurance **shall** advise the eligible worker of the following resources: (1) Five (5) additional weeks of benefits... [under] KRS 341.385(5)... (2) The Work Ready Kentucky Scholarship Program... (3) The Federal Pell Grant Program; (4) The Free Application for Federal Student Aid (FAFSA); and (5) Additional education and training resources..." [278] | **Reemployment Integration:** Leverages the claim notification event to actively counsel claimants on educational opportunities and scholarships to encourage retraining [278]. |
|
|
||||||
| **7** | **Determine Insured Status & Notify Parties** | **KRS 341.410:** "The secretary... **shall**, upon request, determine the insured status of a worker. If a worker is found to have fully insured status... the Office of Unemployment Insurance **shall** notify all interested parties. If found to be not fully insured, the division **shall** notify the worker." [403] | **Insured Status:** Governs basic monetary entitlement determination based on base-period wages [403]. |
|
|
||||||
| **8** | **Cooperation Under the Interstate Benefit Payment Plan** | **787 KAR 1:150 Section 2:** "The secretary **shall** apply the terms of the interstate benefit payment plan in administrative cooperation with other states that have similar administrative provisions..." [96] | **Interstate Claimants:** Establishes cross-border administrative duties to ensure claimants moving out-of-state can access benefits earned in Kentucky [96]. |
|
|
||||||
| **9** | **Mandatory Verification of Child Support Obligations** | **KRS 341.392(1):** "A worker filing a new claim... **shall**... disclose whether or not he owes child support... If [so]... and is determined to be eligible... the secretary **shall** notify the state or local child support enforcement agency enforcing such obligation..." [398] | **Inter-Agency Notice:** Triggers child support notification procedures [398]. |
|
|
||||||
| **10** | **Mandated Child Support Deductions** | **KRS 341.392(2):** "The secretary **shall** deduct and withhold from any unemployment benefits payable to a worker that owes child support obligations [the amount specified, agreed upon, or served via legal process]." [398] | **Priority of Deductions:** Child support deductions occupy a top mandatory priority under priority-ordering rules [158, 398]. |
|
|
||||||
| **11** | **Federal/State Tax Advisory Notice** | **KRS 341.395(1):** "An individual filing a new claim... **shall**, at the time of filing the claim, be advised that: (a) Unemployment compensation is subject to federal and state income tax; (b) Requirements exist pertaining to estimated tax payments; [and] (c) The individual may elect to have federal income tax deducted..." [400] | **Tax Notice:** Essential tax advisory requirements [400]. |
|
|
||||||
| **12** | **Strict Priority Ordering of Deductions** | **787 KAR 1:320 Section 1:** "If deductions and withholding from benefit payments are required under more than one (1) statute... the priority for deduction **shall** be [ordered: (1) wages/pensions, (2) previous overpayments, (3) child support, (4) other mandatory, (5) voluntary taxes]." [158] | **Priority Protections:** Legally protects the sequence in which claimant checks are reduced [158]. |
|
|
||||||
| **13** | **Establish Procedures for Joint / Group Accounts** | **KRS 341.281(2):** "...Upon his approval of the application, the secretary **shall** establish a group account for [sharing costs among nonprofit or governmental entities]... The secretary **shall** prescribe such procedures as he deems necessary with respect to applications..." [346] | **Cost Sharing:** Group representative serves as the agent [346]. |
|
|
||||||
| **14** | **Application of Statistical Profiling Model** | **787 KAR 1:310 Section 2(3):** "The secretary **shall** utilize a statistical model of worker profiling as the basis for the identification of claimants for referral for reemployment services." [154] | **Non-Discriminatory Profiling:** Factors used **shall not** include the claimant's age, gender, race, ethnicity, or national origin [154]. |
|
|
||||||
| **15** | **Profiling Referral Limitation** | **787 KAR 1:310 Section 2(5):** "A claimant who is not referred for services within four (4) weeks after identification by the profiling system **shall not** be referred and **shall** be considered to have satisfied the requirements... for the receipt of benefits." [155] | **Statutory Release:** Prevents long-delayed reemployment service mandates from impacting ongoing benefit payments [155]. |
|
|
||||||
| **16** | **Mandatory Rescission of Unlawful Recall Prospects** | **KRS 341.350(7)(c):** "The secretary **shall not** confirm a bona fide return-to-work... prospect if... contrary to federal law... The secretary **shall** rescind any previous confirmation... if [such grounds exist]." [375] | **Conformity:** Protects Kentucky's compliance with federal rules [375]. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. Notices, Tax Determinations, & Employer Accounts
|
|
||||||
|
|
||||||
These duties govern employer status determinations, experience ratings, annual tax notice scheduling, PEO-specific transactions, and statutory requirements for tax collection and levy releases.
|
|
||||||
|
|
||||||
| # | Duty | Source & Requirement | Notes & Operational Context |
|
|
||||||
|---|---|---|---|
|
|
||||||
| **17** | **Prompt Liability Status Notification** | **787 KAR 1:040 Section 1:** "The Office of Unemployment Insurance **shall** promptly notify an employing unit of any determination as to its liability as a subject employer." [15] | **Status Determinations:** Commences the legal period within which an employer can protest liability [15, 69]. |
|
|
||||||
| **18** | **Furnish Employer Notification Posters** | **787 KAR 1:040 Section 2:** "The office **shall** furnish each subject employer with posters (UI-5.1) informing the workers that the employer is a subject employer... and of their potential rights..." [15] | **Workplace Posting:** Mandatory physical posters supplied by the agency at no cost to inform workers of partial benefit rights [15, 402]. |
|
|
||||||
| **19** | **Annual Issuance of Contribution Rate Notices** | **787 KAR 1:210 Section 1(1):** "On or before December 15 of each year, the Office... on behalf of the secretary, **shall** issue to each active employer liable... a Notice of Contribution Rate [Form UI-29]." [118] | **Annual Tax Notification:** The notice **shall** state the active rate schedule, individual account tax details, and the statutory provisions used to calculate the rate [118]. |
|
|
||||||
| **20** | **Mandatory Application of PEO Contribution Election** | **787 KAR 1:370 Section 2(1):** "The Office **shall** apply the PEO's election to all clients covered under the PEO agreement as of the effective date of the election." [171] | **Co-Employer Reporting:** Binds the PEO and clients to a single reporting method (PEO-level or client-level) [171, 172]. |
|
|
||||||
| **21** | **Mandatory Transfer of PEO Client Experience History** | **787 KAR 1:370 Section 2(3)(b):** "The client's benefit experience history... **shall** be transferred by the Office to the account assigned to that client as co-employer, as required by KRS 336.248(1)(b)3.b." [172] | **Experience Ratings:** This transfer **shall** occur without regard to common ownership, management, or control [175]. |
|
|
||||||
| **22** | **Mandatory Proportional Transfer of Successorship Experience** | **KRS 341.540(5)(a):** "Notwithstanding... any successor to a portion of the trade or business... **shall** assume the resources and liabilities of the predecessor's reserve account in proportion to the percentage of the payroll or employees assignable to the transferred portion." [452] | **Proportional Transfer:** Outlines calculation formulas using the last four completed calendar quarters [452]. |
|
|
||||||
| **23** | **Voiding of Proportional Successor Accounts on Layoffs** | **KRS 341.540(5)(b):** "...if [a successor] lays off or terminates more than one-half (1/2) of the original employees transferred within six (6) months... then the succession and creation of the new employing unit **shall** be voided, and the benefits... **shall** be charged to the reserve account of the original employing unit." [452] | **SUTA Dumping Shield:** Prevents artificial tax rate manipulation by spinning off liabilities into shell corporations [452]. |
|
|
||||||
| **24** | **Identify Business Transfers** | **KRS 341.540(9)(a):** "The secretary **shall** establish procedures to identify the transfer of a business for purposes of this section." [454] | **Procedural System:** Active detection system for commercial successorships [454]. |
|
|
||||||
| **25** | **Assess Surcharge Rates When Trust Fund is Insufficient** | **KRS 341.611(3):** "In the event there are insufficient funds... to pay the interest on [Title XII] advances... all subject contributing employers **shall** be assessed a surcharge as provided in KRS 341.614." [467] | **Trust Fund Solvency:** Mandatory surcharge triggered during borrowing periods [467, 472]. |
|
|
||||||
| **26** | **Adjust Surcharge Percentages Annually** | **KRS 341.614(2)(b):** "...the secretary **shall** adjust the surcharge percentage rate based on any increase to the taxable wage base for that calendar year... The purpose... **shall** be to maintain costs per worker comparable to the original surcharge." [472] | **Indexing:** Must correspond with the taxable wage base under KRS 341.030 [472]. |
|
|
||||||
| **27** | **Mandatory Release of Property Levies** | **787 KAR 1:250 Section 1:** "The secretary or his designated representative **shall** release a levy if [employer enters escrow arrangement, furnishes equivalent bond, pays interest in property, signs partial payment, or if the value is insufficient to cover sale costs]." [134] | **Levy Releases:** Restricts the agency from holding property under seize if statutory collection alternatives are satisfied [134, 135]. |
|
|
||||||
| **28** | **Notice of Intent to Levy** | **KRS 341.805(1):** "Levy may be made with respect to any unpaid contributions... only after the cabinet has given notice and demand to the subject employer in writing of the intention... no less than ten (10) days before the date of levy." [499] | **Notice Protections:** Provides the employer with a 10-day warning before physical seizure of property, wages, or assets [499]. |
|
|
||||||
| **29** | **Continuous Wage Levies & Prompt Release Notifications** | **KRS 341.805(2)(b):** "...the cabinet **shall** promptly release the levy when the liability... is satisfied or becomes unenforceable... and **shall** promptly notify the person upon whom such levy was made..." [499] | **Debtor Release:** Requires immediate notification to the employer and bank upon satisfaction of liability to release frozen funds [499]. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Appeals, Hearings, & Due Process Protections
|
|
||||||
|
|
||||||
These represent core due process mandates that referees, hearing officers, and commissioners must follow to ensure fair, impartial administrative hearings.
|
|
||||||
|
|
||||||
| # | Duty | Source & Requirement | Notes & Operational Context |
|
|
||||||
|---|---|---|---|
|
|
||||||
| **30** | **Appoint Impartial Referees** | **KRS 341.420(1):** "The secretary **shall** appoint one (1) or more impartial referees according to KRS 341.125 to hear and decide appealed claims." [429] | **Ref referees:** Provides the first tier of administrative appeal [429]. |
|
|
||||||
| **31** | **Disqualification of Referees & Commissioners** | **787 KAR 1:110 Section 2(3)(a):** "A referee **shall not** participate in the hearing of an appeal in which the referee has an interest." [56] See also **KRS 341.440(1):** "No examiner, referee, or member of the commission **shall** participate in any hearing in which he is an interested party." [433] | **Conflict of Interest:** Basic due process protection preventing agency officials from deciding cases in which they have a personal or financial stake [56, 433]. |
|
|
||||||
| **32** | **Prompt Scheduling & Mailing of Notices** | **787 KAR 1:110 Section 2(2)(a):** "...the Office of Unemployment Insurance **shall** schedule all hearings promptly and **shall** mail notices to the parties specifying the date, time, and place... at least ten (10) days prior to the hearing date." [55] | **Mailing Deadline:** Establishes a mandatory 10-day notice minimum unless parties explicitly waive the timeline [55, 70]. |
|
|
||||||
| **33** | **Mandatory Rescheduling for Good Cause** | **787 KAR 1:110 Section 2(2)(c):** "The office **shall** reschedule the hearing upon presentation by a party of good cause." [56] | **Hearing Accommodations:** Good cause includes current employment conflicts, medical emergencies, family deaths, or acts of God [56, 71]. |
|
|
||||||
| **34** | **Electronic Scheduling (Teleconference Mandate)** | **787 KAR 1:110 Section 2(4)(d):** "The hearing **shall** be scheduled via teleconference if an in-person hearing would: 1. Create undue expense... 2. Require any party to travel more than fifty (50) miles; 3. Put either party or the referee at personal risk; or 4. Create a security risk..." [58] | **Venue Preference:** Dictates teleconference format over in-person if risk, distance, or financial barriers exist [58, 74]. |
|
|
||||||
| **35** | **Informality of Administrative Proceedings** | **787 KAR 1:110 Section 5(4)(a):** "All hearings **shall** be conducted informally without regard to common law, statutory or technical rules... and in a manner as to determine the substantial rights of the parties." [81] | **Relaxed Evidence Rules:** Relaxes common law rules of evidence to ensure unrepresented claimants and employers can present cases [81, 433]. |
|
|
||||||
| **36** | **All Testimony Taken Under Oath** | **787 KAR 1:110 Section 5(4)(b):** "The parties and their witnesses **shall** testify under oath or affirmation." [81] | **Credibility:** Mandatory for establishing a formal record and subject to perjury penalties [81, 306, 433]. |
|
|
||||||
| **37** | **Address All Relevant Appeal Issues** | **787 KAR 1:110 Section 5(4)(c):** "All issues relevant to the appeal **shall** be considered and passed upon." [81] | **Exhaustive Review:** Prevents referees from ignoring key issues presented by the parties [81]. |
|
|
||||||
| **38** | **Active Evidentiary Examination by Referee** | **787 KAR 1:110 Section 2(4)(a)2:** "The referee **shall**, if necessary to secure full information on the issues, examine each party who appears and witnesses." [57] | **Inquisitorial Role:** Prevents passive adjudication; referees must actively ask questions to develop the administrative record [57, 72]. |
|
|
||||||
| **39** | **Due Process Rights to Examine External Evidence** | **787 KAR 1:110 Section 2(4)(a)4:** "If additional evidence is taken, all interested parties **shall** be afforded an opportunity of examining and refuting the evidence." [57] | **Rebuttal Protections:** Essential to avoid one-sided evidence entries during appellate reviews [57, 73]. |
|
|
||||||
| **40** | **Action on Written Fact Stipulations** | **787 KAR 1:110 Section 2(4)(b)2:** If parties stipulate facts, the referee **shall**: "a. Decide the appeal on the basis of the stipulation; or b. Schedule a hearing and take further evidence." [58] | **Consent Facts:** Directs referee steps when parties agree to written undisputed facts [58, 73]. |
|
|
||||||
| **41** | **Immediate Stop-Payment Order Upon Overturning Benefits** | **787 KAR 1:110 Section 2(4)(e):** "If the decision is to deny previously awarded benefits either retroactively or forthwith, a stop payment directive **shall** be issued to the office by the referee on the date the decision is mailed..." [60] | **Financial Integrity:** Prevents the ongoing payout of state funds once a referee issues a benefit disqualification [60]. |
|
|
||||||
| **42** | **Service of Commission Appeals to Other Parties** | **787 KAR 1:110 Section 3(1)(a)2:** "The appeal [to the commission] **shall** be mailed by the office to other interested parties." [61] | **Appellate Notice:** Commences the appellee's clock to submit written responses [61, 63]. |
|
|
||||||
| **43** | **Commission Appeals Decided on the Record** | **787 KAR 1:110 Section 3(2)(a)1:** "...all appeals to the commission **shall** be heard upon the records of the office and the evidence and exhibits introduced before the referee." [62] | **Record Review:** Limits the KUIC from introducing random unrecorded evidence unless proper notice is issued [62, 64]. |
|
|
||||||
| **44** | **Notice of Commission Additional Evidence Hearings** | **787 KAR 1:110 Section 3(2)(b)2:** "If additional evidence is necessary... the parties **shall** be notified of the time and place... at least seven (7) days prior..." [64] | **Hearing Timelines:** Provides a mandatory 7-day preparation window for subsequent evidence hearings before the commission [64]. |
|
|
||||||
| **45** | **Written Decisions with Findings of Fact & Conclusions of Law** | **787 KAR 1:110 Section 3(4)(a)1:** "...the commission **shall** issue a written decision, which **shall** affirm... or present a separate finding of facts, decision, and reasons." [65] | **Standard of Orders:** Decisions **shall** be signed by participating members [65, 75]. |
|
|
||||||
| **46** | **Decision by Majority Rule** | **787 KAR 1:110 Section 3(4)(b)1:** "If a decision of the commission is not unanimous, the decision of the majority **shall** control." [66] | **Panel Voting:** Dissenting members **shall** be permitted to file written dissents detailing reasons [66, 76]. |
|
|
||||||
| **47** | **Mailing of Decisions to All Parties** | **787 KAR 1:110 Section 3(4)(c):** "Copies of the decision **shall** be mailed to all interested parties." [67] | **Mailing Mandate:** Restarts the 20-day countdown for judicial review [67, 77, 434]. |
|
|
||||||
| **48** | **Maintain Valid Precedent Digests** | **787 KAR 1:110 Section 3(6)(a):** "The Kentucky Unemployment Insurance Commission **shall** develop, distribute, and maintain a manual or digest containing all precedent decisions currently valid." [68] | **Legal Consistency:** Restricts arbitrary decisions; binding precedents are made accessible to lower determination levels [66, 68]. |
|
|
||||||
| **49** | **Provide Decisions Free of Charge** | **787 KAR 1:110 Section 3(6)(b):** "Individual decisions **shall** be available on request without charge." [69] | **Public Records:** Protects public access to administrative law precedents [69]. |
|
|
||||||
| **50** | **Appoint Impartial Hearing Officers for Tax Credits** | **787 KAR 2:030 Section 4(1):** "...Upon receipt of the written request... the Secretary... **shall** appoint an impartial hearing officer to hear and decide appealed denials." [191] | **Tax Credit Appeals:** Applies to the Unemployment Tax Credit Program under KRS 141.065 [185, 191]. |
|
|
||||||
| **51** | **Record Tax Credit Hearings** | **787 KAR 2:030 Section 4(5):** "All testimony at the hearing **shall** be recorded." [194] | **Evidentiary Record:** Establishes the administrative record for subsequent appeals [194]. |
|
|
||||||
| **52** | **Issue Timely Decisions on Tax Credits** | **787 KAR 2:030 Section 4(7):** "Within thirty (30) days of adjournment... the hearing officer **shall** make written findings of fact, conclusions of law, and a final decision..." [194] | **Decision Timelines:** Imposes a strict 30-day decision deadline [194]. |
|
|
||||||
| **53** | **Commission Orders Following Judicial Review** | **KRS 341.450(5):** "Upon the final determination of such judicial proceeding, the commission **shall** enter an order in accordance with such determination." [434] | **Judicial Enforcement:** Mandates prompt administrative implementation of court orders [434]. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Program Integrity, Fraud Prevention, & Auditing
|
|
||||||
|
|
||||||
These duties mandate extensive database cross-matches, identity verification protocols, overpayment waivers, and weekly auditing procedures to protect state funds.
|
|
||||||
|
|
||||||
| # | Duty | Source & Requirement | Notes & Operational Context |
|
|
||||||
|---|---|---|---|
|
|
||||||
| **54** | **Weekly Directory of New Hires (NDNH) Cross-Checks** | **KRS 341.414(1)(a) & (c):** "...the office **shall**: (a) Use state or commercially available databases to check new hire records against the state's unemployment insurance rolls on a weekly basis; [and] (c) Check new hire records against the National Directory of New Hires on a weekly basis;" [409] | **Work-Search Audits:** Identifies claimants claiming benefits while concurrently receiving payroll wages [409]. |
|
|
||||||
| **55** | **Weekly Incarceration Cross-Checks** | **KRS 341.414(1)(d):** "Check the unemployment insurance rolls against the Kentucky Department of Corrections list of incarcerated individuals on a weekly basis;" [409] | **Incarceration Audits:** Implements weekly database cross-checks to ensure benefits are not paid to inmates [409]. |
|
|
||||||
| **56** | **Establish OIG Anti-Fraud Agreements** | **KRS 341.414(1)(e):** "Enter into a cooperative agreement with the United States Department of Labor's Office of Inspector General to proactively detect and investigate cases of unemployment fraud;" [409] | **Federal Integration:** Coordinates federal/state enforcement resources [409]. |
|
|
||||||
| **57** | **Mandatory Eligibility Review on Change of Circumstance** | **KRS 341.414(1)(f):** "Review the individual's case and make a final determination of his or her eligibility when the office receives information that an individual... has had a change in circumstances..." [409] | **Eligibility Re-evaluations:** Triggered by employer refusals of work or payroll alerts [389, 409]. |
|
|
||||||
| **58** | **Promulgate Overpayment Recovery Rules** | **KRS 341.414(1)(h):** "Promulgate, adopt, and implement internal administrative regulations and policies to recover improper overpayments of unemployment benefits;" [409] | **Recovery Policies:** Internal rules for recovery and recoupment [409]. |
|
|
||||||
| **59** | **Mandatory Multi-Factor Authentication** | **KRS 341.414(1)(i):** "Require at least two-factor authentication from every benefit applicant prior to payment of any benefit;" [409] | **Identity Auditing:** Establishes two-factor authentication as an absolute prerequisite to fund disbursement [405, 409]. |
|
|
||||||
| **60** | **Maintain Waiver Records** | **KRS 341.414(1)(j):** "Maintain records of any and all applications for waiver of overpayment." [409] | **Waiver Records:** Promotes statutory tracking of overpayment waiver requests [409, 410]. |
|
|
||||||
| **61** | **Supply Attorney General with Records** | **KRS 341.414(2):** "The office **shall** provide the Office of the Attorney General with all records it requests for the purposes of investigating and prosecuting fraudulent unemployment insurance activity." [409] | **Prosecutorial Cooperation:** Assures records are immediately transferred for criminal enforcement [409]. |
|
|
||||||
| **62** | **Conduct Weekly Randomized Claim Audits** | **KRS 341.350(10):** "The cabinet **shall** conduct randomized weekly audits of a number determined by the secretary as sufficient to evaluate compliance with the work search activity requirements..." [377] See also **787 KAR 1:090 Section 7:** "The secretary **shall** conduct random audits of claims." [49] | **Work-Search Audits:** Evaluates claimant compliance with the five weekly work-search requirement [40, 372, 377]. |
|
|
||||||
| **63** | **Refer Suspected Fraud within Thirty (30) Days** | **KRS 341.416(1)-(2):** "Any person suspected of fraudulently obtaining... **shall** be referred by the cabinet to the appropriate county attorney or Commonwealth's attorney and the United States Department of Labor... no later than thirty (30) days after determining suspected fraud..." [415] | **Fraud Referrals:** Imposes a strict 30-day timeline to compile and forward case files to state and federal prosecutors [415]. |
|
|
||||||
| **64** | **Waiver of Non-Fault Overpayments** | **787 KAR 1:360 Section 2:** "The secretary **shall** issue a waiver of the overpayment if the secretary determines that: (1) The overpayment was made... without fault... and (2) Recovery would be contrary to equity and good conscience..." [163] See also **KRS 341.413(1):** "The secretary **shall** waive an overpayment of benefits [for CARES claims]..." [407] | **No-Fault Overpayments:** Overpayments caused solely by office error or auto-payment must be waived if recovery causes financial hardship [163, 164]. |
|
|
||||||
| **65** | **Confidentiality Regulations** | **787 KAR 3:020 (KRS 151B.280):** "...requires the secretary... to promulgate administrative regulations to protect the confidential nature of all records and reports which directly or indirectly identify a client..." [237] | **Records Protection:** Protects employment records from public disclosure [237, 302]. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Fund Management & Financial Controls
|
|
||||||
|
|
||||||
These duties establish strict fiduciary standards, fund deposit rules, requisition protocols, and benefit protection guidelines to safeguard the State Treasury.
|
|
||||||
|
|
||||||
| # | Duty | Source & Requirement | Notes & Operational Context |
|
|
||||||
|---|---|---|---|
|
|
||||||
| **66** | **Immediate Clearing Account Transfers** | **787 KAR 1:140 Section 1:** "All transfers to the state's account in the Unemployment Trust Fund or refund payments made from the clearing account **shall** be made by the treasurer of the unemployment insurance fund immediately upon receipt of a written order..." [89] | **Transfer Speed:** Fiduciary requirement to immediately transfer cleared revenue to the federal treasury [89, 442]. |
|
|
||||||
| **67** | **24-Hour Benefit Requisitions** | **787 KAR 1:140 Section 2:** "Requisitions from the state's account... for claimant benefits or employer refunds **shall** be made by the treasurer within twenty-four (24) hours after the receipt of a written order..." [89] | **Claim Payout Speed:** Mandates that once the OUI orders funds, the Treasurer has a maximum of 24 hours to execute the federal withdrawal [89]. |
|
|
||||||
| **68** | **Immediate Separation & Deposit of Withdrawn Funds** | **787 KAR 1:140 Section 2:** "...Withdrawals for benefits, **shall** immediately be deposited in the benefit account. Withdrawals for employer refunds **shall** immediately be deposited in the clearing account." [89] See also **KRS 341.510(3):** "...deposit the money in the benefit account and **shall** issue his vouchers for the payment... solely from the benefit account." [443] | **Account Integrity:** Dictates strict segregation of operational benefit accounts from clearing tax accounts [89, 442, 443]. |
|
|
||||||
| **69** | **Prepare & Certify Benefit Payment Registers** | **787 KAR 1:140 Section 3(1):** "A benefit payment register... **shall** be prepared and presented to the treasurer and certified as correct... [showing claimant name, SSN, payment amount, compensable period, issuance date, program type, and earnings]." [90] | **Financial Integrity:** Establishes the prerequisite audit trail showing seven statutory metrics before any state checks are cut [90, 91]. |
|
|
||||||
| **70** | **Affix Signature and Written Executive Approval on Vouchers** | **KRS 341.510(3):** "All vouchers issued by the Treasurer for the payment of benefits and refunds **shall** bear the signature of the Treasurer and the approval in writing of the secretary of the Finance and Administration Cabinet." [443] | **Co-Signature Controls:** Imposes high-level co-signature requirement for public fund disbursements [443]. |
|
|
||||||
| **71** | **Maintain Unemployment Insurance Fund Separate from General Fund** | **KRS 341.490(1):** "...a special fund known as the unemployment insurance fund which **shall** be administered separate and apart from all public money or funds of this state." [441] | **Segregated Public Funds:** Protects the state unemployment fund from diversion to the state's General Fund [441]. |
|
|
||||||
| **72** | **Deposit All Collected Revenues into the Clearing Account** | **KRS 341.500(2):** "All money payable to the fund, upon receipt... **shall** be forwarded to the Treasurer, who **shall** immediately deposit it in the clearing account." [442] | **Deposit Flow:** Dictates initial landing zone for all inbound payments [442]. |
|
|
||||||
| **73** | **Pay Allowed Claims Promptly Despite Pending Appeal (Double Affirmation Rule)** | **KRS 341.420(3):** "If benefits are allowed by a determination... or a decision... such benefits **shall** be paid promptly without regard to the pendency of an appeal... No injunction, supersedeas, stay or other writ... suspending payment... **shall** be issued." [429] | **Anti-Delay Protection:** Prevents employers or subsequent appeals from holding up benefits once twice-approved [429]. |
|
|
||||||
| **74** | **No Vested Right to Prevent Legislative Adjustments** | **KRS 341.600(2):** "No vested right **shall** exist against amendment or repeal of any part of this chapter." [465] | **Statutory Flexibility:** Allows the legislature to amend tax rates and weekly benefit scales [465]. |
|
|
||||||
| **75** | **Sole Use of Federal Monies for Administrative Costs** | **KRS 341.620:** "All moneys received pursuant to Title III of the Social Security Act... **shall** be expended solely for the purposes and in the amounts found necessary by the Federal Bureau of Employment Security..." [475] | **Administrative Allocation:** Strict spending limitations on federal administrative grants [475]. |
|
|
||||||
| **76** | **Enforce Delinquency Civil Actions Without Jury Interventions** | **KRS 341.300(2):** "...Civil actions brought under this section **shall** be heard by the court, without the intervention of a jury, at the earliest possible date..." [355] See also **KRS 341.614(6):** "...Surcharge... **shall** be heard by the court, without the intervention of a jury..." [473] | **Accelerated Actions:** Establishes priority civil proceedings for delinquent UI tax collections [355, 473]. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. Legislative Reporting & Federal Conformity
|
|
||||||
|
|
||||||
These duties enforce accountability through mandatory reporting timelines to the Governor, the Legislative Research Commission (LRC), and the Interim Joint Committees.
|
|
||||||
|
|
||||||
| # | Duty | Source & Requirement | Notes & Operational Context |
|
|
||||||
|---|---|---|---|
|
|
||||||
| **77** | **LRC Notification of Federal Conformity Issues** | **KRS 341.126:** "The secretary **shall** provide to the Legislative Research Commission a copy of any notification received... regarding the conformity of state unemployment compensation... within five (5) days of receiving the notification." [285] | **Conformity Notices:** Crucial for federal tax credits and administrative funding [285]. |
|
|
||||||
| **78** | **LRC Annual Improper Payments & Recovery Report** | **KRS 341.414(3):** "The office **shall** complete an annual report for the Legislative Research Commission which details efforts to recover improper benefit payments... summaries of all cases where repayment was not attempted... no later than October 1 of each year." [409] | **Oct 1 Deadline:** Requires a transparent annual summary of improper payments, waivers granted, and actual recovery rates [409]. |
|
|
||||||
| **79** | **Annual Trust Fund Solvency and Economic Report** | **KRS 341.127(1):** "By December 1... the cabinet **shall** report to the Governor and the Interim Joint Committee...: (a) The status of the unemployment trust fund... (b) The average claim duration... [and] (c) The average weekly wage for individuals receiving unemployment benefits." [287] | **Dec 1 Solvency Report:** Expired on January 31, 2026 [288], but represents a critical historical requirement for fiscal planning [287]. |
|
|
||||||
| **80** | **Shared Work Benefit Annual Impact Report** | **KRS 341.127(3):** "By December 1... the cabinet **shall** report to the Governor and the Interim Joint Committee... and provide analysis of the impact of the shared work benefits..." [287] | **Dec 1 Shared-Work Report:** Evaluates the impact of shared-work plans on employer taxes and the Trust Fund [287]. |
|
|
||||||
| **81** | **Submit Annual Recommendations to the Governor** | **KRS 341.125(4):** "The secretary **shall** submit to the Governor an annual report covering the administration and operation of this chapter and make such recommendations for amendments..." [284] | **Annual Legislative Proposals:** Governs the formal path to propose legislative updates [284]. |
|
|
||||||
| **82** | **Annually Update the Mercer Computerized Model** | **KRS 341.116:** "It shall be the responsibility of the secretary to annually update and maintain the unemployment insurance mercer computerized model and to make this model available to the Legislative Research Commission..." [282] | **Mercer Simulation Model:** Ensures legislative access to financial projection models [282]. |
|
|
||||||
| **83** | **Annually Compute the Insured Average Weekly Wage** | **KRS 341.380(3):** "Prior to the first day of July of each year the secretary **shall** determine the average weekly wage for insured employment by dividing [total reported wages by average monthly employment]..." [391] | **July 1 Deadline:** Mathematically establishes the maximum weekly benefit rate for the next fiscal year [391]. |
|
|
||||||
| **84** | **Mandatory Extended Benefit Announcements** | **KRS 341.740:** "Whenever an extended benefit period is to become effective... or... terminated... the secretary **shall** make an appropriate public announcement." [495] | **State "On" or "Off" Triggers:** Establishes public notice mandates for extended benefit periods [273, 495]. |
|
|
||||||
| **85** | **Mandatory Written Notices of Seized Property** | **KRS 341.800(2):** "As soon as practicable after seizure... notice in writing **shall** be given by the secretary or his delegate to the owner... either in person or by certified mail... specifying the sum demanded..." [496] | **Post-Seizure Notices:** Post-seizure property inventory and valuation notice [496]. |
|
|
||||||
| **86** | **Notice of Public Property Sale** | **KRS 341.800(3):** "The secretary... **shall**... cause a notification of the sale... to be published in the newspaper with the largest circulation within the county... once each week for three (3) successive weeks... [and] posted at the courthouse..." [496] | **Sale Publication:** Essential procedural mandate before conducting a tax foreclosure sale [496]. |
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
```
|
|
||||||
jennifer 14:03
|
|
||||||
ELC UI Appeals <UIappeals@ky.gov>, greg.higgins@ky.gov, Hockensmith, Sabrena (ELC) <Sabrena.Hockensmith@ky.gov>, Boggs, Leah C (ELC) <lboggs@ky.gov>, ELC Office of Policy & Audit <elc.opa@ky.gov>, UIcommission@ky.gov, Rodgers Johnson, Rebecca A (ELC) <rebecca.rodgers@ky.gov>, oui.cdo.openrecords@ky.gov, jamie.link@ky.gov, governor.constituentservices@ky.gov, Doucet, Larry B (ELC) <larry.doucet@ky.gov>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
```
|
|
||||||
@@ -0,0 +1,527 @@
|
|||||||
|
# 🔗 AI Handoff — Projects & Agent Workspace
|
||||||
|
|
||||||
|
**Live at:** [https://app.ai-handoff.work](https://app.ai-handoff.work)
|
||||||
|
**Repo:** [https://github.com/stateofshit/url-shortener](https://github.com/stateofshit/url-shortener)
|
||||||
|
**Built:** August 12, 2026
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## What This Covers
|
||||||
|
|
||||||
|
### 1. URL Shortener (Production App)
|
||||||
|
A **production-ready URL shortener** with **live click analytics**, deployed entirely on **Cloudflare's edge infrastructure**. No accounts, no databases to manage, no servers to maintain. Just paste a long URL, get a short one, and watch the clicks roll in.
|
||||||
|
|
||||||
|
📄 See below for full URL Shortener documentation.
|
||||||
|
|
||||||
|
### 2. Assistant Workspace (Agent Project)
|
||||||
|
An **AI agent starter** built with Cloudflare Agents SDK + TypeScript, living at:
|
||||||
|
|
||||||
|
```
|
||||||
|
/home/user/03-projects/assistant-workspace/
|
||||||
|
```
|
||||||
|
|
||||||
|
Copied from `.agents/skills/agents-sdk/my-agent/`. Purpose-built for autonomous agent behavior using the Cloudflare Workers platform.
|
||||||
|
|
||||||
|
#### Quick Start
|
||||||
|
```bash
|
||||||
|
cd /home/user/03-projects/assistant-workspace
|
||||||
|
npx wrangler dev # local dev
|
||||||
|
npx wrangler deploy # production
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Structure
|
||||||
|
```
|
||||||
|
assistant-workspace/
|
||||||
|
├── AGENTS.md # Agent guidelines & docs
|
||||||
|
├── src/ # Agent source code
|
||||||
|
├── test/ # Vitest tests
|
||||||
|
├── wrangler.jsonc # Worker bindings config
|
||||||
|
└── package.json # Agents SDK deps
|
||||||
|
```
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🏗️ Architecture Overview (URL Shortener)
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ YOUR BROWSER │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Cloudflare DNS (ai-handoff.work) │
|
||||||
|
│ shit.ai-handoff.work → Worker API │
|
||||||
|
│ dash.ai-handoff.work → Dashboard (Pages) │
|
||||||
|
│ app.ai-handoff.work → React App (Pages) │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Cloudflare Workers + Pages + D1 │
|
||||||
|
│ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Worker: url-shortener-api (shit.ai-handoff.work) │ │
|
||||||
|
│ │ - POST /api/shorten │ │
|
||||||
|
│ │ - GET /api/links │ │
|
||||||
|
│ │ - GET /api/trends/:code │ │
|
||||||
|
│ │ - GET /{code} → 301 redirect to original URL │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Pages: shrtlink (dash.ai-handoff.work) │ │
|
||||||
|
│ │ - Static HTML/CSS/JS dashboard with live stats │ │
|
||||||
|
│ │ - Copy-link button, 14-day sparkline trends │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Pages: shrtlink-app (app.ai-handoff.work) │ │
|
||||||
|
│ │ - React Vite app (same functionality, different UI) │ │
|
||||||
|
│ │ - Built with React 19 + Vite 6 │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ D1 Database: url-shortener-db │ │
|
||||||
|
│ │ - urls table: code, url, clicks, created_at │ │
|
||||||
|
│ │ - clicks table: code, clicked_at (per-click history) │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🛠️ Tech Stack (URL Shortener)
|
||||||
|
|
||||||
|
| Layer | Technology | Purpose |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| **Edge Runtime** | Cloudflare Workers | URL shortening, redirects, API endpoints |
|
||||||
|
| **Static Sites** | Cloudflare Pages | Dashboard + React app deployment |
|
||||||
|
| **Database** | Cloudflare D1 (SQLite) | Store URLs, click counts, click timestamps |
|
||||||
|
| **Frontend (Dashboard)** | Vanilla HTML/CSS/JS | Lightweight, fast, no build step |
|
||||||
|
| **Frontend (React App)** | React 19 + Vite 6 | Modern SPA with same features |
|
||||||
|
| **CI/CD** | GitHub Actions | Auto-deploy on `git push master` |
|
||||||
|
| **Custom Domains** | Cloudflare DNS | `*.ai-handoff.work` |
|
||||||
|
|
||||||
|
**Agent Workspace uses:** Cloudflare Workers + Agents SDK + TypeScript + Vitest
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🌐 Custom Domains (URL Shortener)
|
||||||
|
|
||||||
|
All domains point to Cloudflare infrastructure and are **proxied** (CDN + security enabled):
|
||||||
|
| Domain | Target | Status |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `shit.ai-handoff.work` | Worker API | ✅ Live (200) |
|
||||||
|
| `dash.ai-handoff.work` | Dashboard (Pages `shrtlink`) | ✅ Live (200) |
|
||||||
|
| `app.ai-handoff.work` | React App (Pages `shrtlink-app`) | ✅ Live (200) |
|
||||||
|
|
||||||
|
**Why custom domains?**
|
||||||
|
|
||||||
|
* Professional URLs (not `workers.dev` or `pages.dev`)
|
||||||
|
* SSL certificates auto-provisioned by Cloudflare
|
||||||
|
* CDN caching at the edge
|
||||||
|
* DDoS protection via Cloudflare
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## ✨ Features (URL Shortener)
|
||||||
|
|
||||||
|
### 1. URL Shortening
|
||||||
|
* Paste any `http://` or `https://` URL
|
||||||
|
* Get a 6-character short code (e.g., `f5RHkl`)
|
||||||
|
* Short URL: `https://shit.ai-handoff.work/f5RHkl`
|
||||||
|
* **Strict validation**: rejects `javascript:`, `ftp://`, etc.
|
||||||
|
* **Dedupe**: shortening the same URL twice returns the existing code
|
||||||
|
|
||||||
|
### 2. Live Dashboard
|
||||||
|
* **Total links count** (real-time from D1)
|
||||||
|
* **Total clicks count** (sum of all link clicks)
|
||||||
|
* **Table of all links** with:
|
||||||
|
* Short URL (clickable)
|
||||||
|
* Destination URL (truncated if >60 chars)
|
||||||
|
* Click count (badge)
|
||||||
|
* 14-day trend sparkline (SVG bar chart)
|
||||||
|
* Creation date
|
||||||
|
* Copy button (⎘ → ✓ with clipboard API)
|
||||||
|
* **Auto-refresh** every 15 seconds
|
||||||
|
|
||||||
|
### 3. Click Analytics
|
||||||
|
* **Per-click tracking**: every redirect logs a timestamp to `clicks` table
|
||||||
|
* **Daily buckets**: `/api/trends/:code` returns `{day, n}` for last 14 days
|
||||||
|
* **Sparkline visualization**: inline SVG bar chart per link
|
||||||
|
* **Zero data loss**: historical clicks preserved even if URL is deleted
|
||||||
|
|
||||||
|
### 4. Developer Experience
|
||||||
|
* **Auto-deploy**: push to `master` → GitHub Actions → Cloudflare
|
||||||
|
* **Local dev**: `npm run dev` (Vite) + `wrangler dev` (Worker)
|
||||||
|
* **Type-safe**: D1 queries via Wrangler bindings
|
||||||
|
* **Testable**: Playwright E2E tests in `/tmp/*.mjs`
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 📁 Project Structure (URL Shortener)
|
||||||
|
|
||||||
|
```
|
||||||
|
url-shortener/
|
||||||
|
├── .github/workflows/
|
||||||
|
│ └── deploy.yml # GitHub Actions CI/CD
|
||||||
|
├── .wrangler/ # Wrangler cache (gitignored)
|
||||||
|
├── dashboard/ # Static dashboard (HTML/CSS/JS)
|
||||||
|
│ ├── index.html
|
||||||
|
│ ├── app.js # Dashboard logic
|
||||||
|
│ └── styles.css # Dark theme styles
|
||||||
|
├── dist/ # Vite build output (gitignored)
|
||||||
|
├── src/ # React app source (Vite)
|
||||||
|
│ ├── main.jsx
|
||||||
|
│ └── App.jsx
|
||||||
|
├── wrangler-api/ # Worker + D1 migrations
|
||||||
|
│ ├── src/
|
||||||
|
│ │ └── index.js # Worker fetch handler
|
||||||
|
│ ├── migrations/
|
||||||
|
│ │ ├── 000_create_urls.sql
|
||||||
|
│ │ ├── 001_add_clicks.sql
|
||||||
|
│ │ └── 002_create_clicks.sql
|
||||||
|
│ ├── wrangler.jsonc # Worker config + D1 binding
|
||||||
|
│ └── package.json
|
||||||
|
├── .env # Cloudflare credentials (gitignored)
|
||||||
|
├── .env.example # Example env vars
|
||||||
|
├── .gitignore
|
||||||
|
├── index.html # Root (redirects to dashboard)
|
||||||
|
├── package.json # Vite + React deps
|
||||||
|
└── vite.config.js
|
||||||
|
```
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🗄️ Database Schema (URL Shortener - D1)
|
||||||
|
|
||||||
|
### `urls` table
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE urls (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
code TEXT NOT NULL UNIQUE, -- 6-char short code
|
||||||
|
url TEXT NOT NULL, -- original URL
|
||||||
|
clicks INTEGER NOT NULL DEFAULT 0, -- running total
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
### `clicks` table (migration 002)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE clicks (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
code TEXT NOT NULL,
|
||||||
|
clicked_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
FOREIGN KEY (code) REFERENCES urls(code)
|
||||||
|
);
|
||||||
|
CREATE INDEX idx_clicks_code ON clicks(code, clicked_at);
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why two tables?**
|
||||||
|
|
||||||
|
* `urls.clicks` = fast counter for the dashboard badge
|
||||||
|
* `clicks` table = per-click history for trend charts
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🔌 API Endpoints (URL Shortener)
|
||||||
|
|
||||||
|
All endpoints on `https://shit.ai-handoff.work`:
|
||||||
|
|
||||||
|
### `POST /api/shorten`
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "url": "https://example.com/very/long/path" }
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response (200):**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"shortUrl": "https://shit.ai-handoff.work/f5RHkl",
|
||||||
|
"code": "f5RHkl",
|
||||||
|
"dedupe": false
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Errors:**
|
||||||
|
* `400` — invalid URL, missing `url` field
|
||||||
|
* `500` — internal error
|
||||||
|
|
||||||
|
### `GET /api/links`
|
||||||
|
|
||||||
|
**Response (200):**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"links": [
|
||||||
|
{
|
||||||
|
"code": "f5RHkl",
|
||||||
|
"url": "https://example.com/long/path",
|
||||||
|
"clicks": 42,
|
||||||
|
"created_at": "2026-08-12T03:00:00Z"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `GET /api/trends/:code`
|
||||||
|
|
||||||
|
**Response (200):**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"code": "f5RHkl",
|
||||||
|
"daily": [
|
||||||
|
{ "day": "2026-08-11", "n": 3 },
|
||||||
|
{ "day": "2026-08-12", "n": 4 }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `GET /{code}` (redirect)
|
||||||
|
|
||||||
|
**Behavior:**
|
||||||
|
* 301 Moved Permanently → original URL
|
||||||
|
* Increments `urls.clicks` + logs to `clicks` table
|
||||||
|
* CORS headers for cross-origin requests
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🚀 Deployment
|
||||||
|
|
||||||
|
### Manual Deploy (URL Shortener)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Worker
|
||||||
|
cd wrangler-api && npx wrangler deploy
|
||||||
|
|
||||||
|
# Dashboard (Pages)
|
||||||
|
cd dashboard && npx wrangler pages deploy ./ --project-name=shrtlink
|
||||||
|
|
||||||
|
# React app (Pages)
|
||||||
|
npx wrangler pages deploy ./dist --project-name=shrtlink-app
|
||||||
|
```
|
||||||
|
|
||||||
|
### Auto-Deploy (GitHub Actions)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# Triggered on: git push origin master
|
||||||
|
# Steps:
|
||||||
|
# 1. Build Vite app (npm run build)
|
||||||
|
# 2. Deploy Worker (npx wrangler deploy)
|
||||||
|
# 3. Deploy Dashboard (npx wrangler pages deploy)
|
||||||
|
# 4. Deploy React app (npx wrangler pages deploy)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Secrets required:**
|
||||||
|
* `CLOUDFLARE_API_TOKEN` — Cloudflare API token with Pages + Workers permissions
|
||||||
|
* `CLOUDFLARE_ACCOUNT_ID` — Cloudflare account ID
|
||||||
|
|
||||||
|
### Agent Workspace Deploy
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/user/03-projects/assistant-workspace
|
||||||
|
npx wrangler deploy
|
||||||
|
```
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🧪 Testing (URL Shortener)
|
||||||
|
|
||||||
|
### Local Development
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Terminal 1: Worker (localhost:8787)
|
||||||
|
cd wrangler-api && npx wrangler dev
|
||||||
|
|
||||||
|
# Terminal 2: Vite dev server (localhost:5173)
|
||||||
|
npm run dev
|
||||||
|
|
||||||
|
# Terminal 3: Dashboard (localhost:3000)
|
||||||
|
cd dashboard && npx wrangler pages dev ./
|
||||||
|
```
|
||||||
|
|
||||||
|
### E2E Tests (Playwright)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Test dashboard features
|
||||||
|
node /tmp/dash-features.mjs
|
||||||
|
|
||||||
|
# Test copy button + trends
|
||||||
|
node /tmp/copy-test.mjs
|
||||||
|
|
||||||
|
# Full end-to-end
|
||||||
|
node /tmp/custom-e2e.mjs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Agent Workspace Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/user/03-projects/assistant-workspace
|
||||||
|
npx vitest
|
||||||
|
```
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🔒 Security (URL Shortener)
|
||||||
|
|
||||||
|
* **Strict URL validation**: only `http:` and `https:` protocols allowed
|
||||||
|
* **CORS**: `Access-Control-Allow-Origin: *` (public API)
|
||||||
|
* **Dedupe**: prevents duplicate rows for same URL
|
||||||
|
* **Collision check**: 6-char codes from 62^6 = ~56B combinations
|
||||||
|
* **SSL**: auto-provisioned by Cloudflare for all custom domains
|
||||||
|
* **DDoS protection**: Cloudflare edge network
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 📊 Live Stats (URL Shortener — as of Aug 12, 2026)
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
| --- | --- |
|
||||||
|
| Total links | 11 |
|
||||||
|
| Total clicks | 8 |
|
||||||
|
| Active domains | 3 |
|
||||||
|
| Deployments | 42s avg |
|
||||||
|
| Uptime | 100% |
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🎯 How to Use
|
||||||
|
|
||||||
|
### As a User (URL Shortener)
|
||||||
|
1. Go to [**https://app.ai-handoff.work**](https://app.ai-handoff.work) or [**https://dash.ai-handoff.work**](https://dash.ai-handoff.work)
|
||||||
|
2. Paste a long URL into the input field
|
||||||
|
3. Click **Shorten**
|
||||||
|
4. Copy the short URL (⎘ button) or click to open it
|
||||||
|
5. Share it — when people click it, you'll see the click count increase in real-time
|
||||||
|
|
||||||
|
### As a Developer (URL Shortener)
|
||||||
|
1. Clone the repo: `git clone https://github.com/stateofshit/url-shortener`
|
||||||
|
2. Set up Cloudflare credentials in `.env` (see `.env.example`)
|
||||||
|
3. Run locally: `npm run dev` + `cd wrangler-api && npx wrangler dev`
|
||||||
|
4. Deploy: `git push origin master` (auto-deploys via GitHub Actions)
|
||||||
|
|
||||||
|
### As a Developer (Agent Workspace)
|
||||||
|
1. Workspace lives at: `/home/user/03-projects/assistant-workspace/`
|
||||||
|
2. Source synced from: `.agents/skills/agents-sdk/my-agent/`
|
||||||
|
3. Run locally: `cd assistant-workspace && npx wrangler dev`
|
||||||
|
4. Deploy: `cd assistant-workspace && npx wrangler deploy`
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## Key Design Decisions (URL Shortener)
|
||||||
|
|
||||||
|
### Why Cloudflare?
|
||||||
|
* **Edge runtime**: Workers execute at the edge (closest to user)
|
||||||
|
* **Free tier**: 100K requests/day on Workers, 10K on D1
|
||||||
|
* **Zero config**: No servers, no scaling, no DevOps
|
||||||
|
* **Built-in SSL**: Automatic HTTPS for custom domains
|
||||||
|
|
||||||
|
### Why D1 (SQLite) instead of KV?
|
||||||
|
* **SQL queries**: need `GROUP BY`, `ORDER BY`, `JOIN` for analytics
|
||||||
|
* **Relational**: `clicks` table references `urls` table
|
||||||
|
* **Cost**: D1 is cheaper than Vectorize for this use case
|
||||||
|
|
||||||
|
### Why two frontends?
|
||||||
|
* **Dashboard** (vanilla): lightweight, fast, no build step
|
||||||
|
* **React app** (Vite): modern SPA, same features, different UI
|
||||||
|
* **Proof of concept**: shows both approaches work on Cloudflare
|
||||||
|
|
||||||
|
### Why 6-character codes?
|
||||||
|
* **Short**: `f5RHkl` is memorable
|
||||||
|
* **Collision-resistant**: 62^6 = 56 billion combinations
|
||||||
|
* **URL-friendly**: no special characters, no encoding needed
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🚧 Known Limitations (URL Shortener)
|
||||||
|
|
||||||
|
1. **No user accounts**: anyone with the link can see the dashboard
|
||||||
|
2. **No custom codes**: codes are randomly generated (no `bit.ly`-style custom slugs)
|
||||||
|
3. **No link expiration**: links never expire (can add TTL later)
|
||||||
|
4. **No QR codes**: short URLs are text only (can add QR generation)
|
||||||
|
5. **No mobile app**: web-only (can add PWA later)
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🔮 Future Enhancements
|
||||||
|
|
||||||
|
### High Priority (URL Shortener)
|
||||||
|
- [ ] **Cloudflare Access**: lock `dash.ai-handoff.work` behind login
|
||||||
|
- [ ] **Custom codes**: allow users to set their own short URL
|
||||||
|
- [ ] **Link groups**: organize links into campaigns
|
||||||
|
- [ ] **Export data**: download click analytics as CSV/JSON
|
||||||
|
|
||||||
|
### Medium Priority (URL Shortener)
|
||||||
|
- [ ] **QR codes**: generate QR for each short URL
|
||||||
|
- [ ] **UTM tracking**: append `?utm_source=twitter` to destination
|
||||||
|
- [ ] **Link expiration**: set TTL
|
||||||
|
- [ ] **Click geo**: log country/region per click
|
||||||
|
|
||||||
|
### Low Priority (URL Shortener)
|
||||||
|
- [ ] **Mobile PWA**: installable app with offline support
|
||||||
|
- [ ] **Browser extension**: right-click → "Shorten this URL"
|
||||||
|
- [ ] **API keys**: rate-limit per user
|
||||||
|
- [ ] **Webhooks**: notify on new click
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 📝 Git History (Key Commits — URL Shortener)
|
||||||
|
|
||||||
|
| Commit | Message | Date |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `ab16107` | feat: strict URL validation, dedupe, click analytics; add landing page + live dashboard | Aug 11 |
|
||||||
|
| `c98074` | chore: point frontends at custom API domain shit.ai-handoff.work | Aug 12 |
|
||||||
|
| `c66bd15` | chore: deploy vite app as shrtlink-app Pages project on app.ai-handoff.work | Aug 12 |
|
||||||
|
| `48481af` | feat: add copy-link button + 14-day click trend sparklines to dashboard | Aug 12 |
|
||||||
|
| `0562e49` | ci: use npx wrangler for all deployments | Aug 12 |
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 💡 Lessons Learned (URL Shortener)
|
||||||
|
|
||||||
|
1. **Custom domains need DNS records**: Cloudflare Pages doesn't auto-create CNAME for all subdomains — had to manually add `app → shrtlink-app.pages.dev`
|
||||||
|
2. **D1 migrations are idempotent**: `CREATE TABLE IF NOT EXISTS` is safe to run multiple times
|
||||||
|
3. **Click logging is cheap**: D1 is SQLite, so logging every click is fine
|
||||||
|
4. **Sparklines are easy**: inline SVG with `<rect>` elements, no chart library needed
|
||||||
|
5. **GitHub Actions secrets are mandatory**: workflow fails silently if secrets are missing
|
||||||
|
6. **Wrangler CLI is powerful**: `npx wrangler pages deploy` works from any directory
|
||||||
|
|
||||||
|
## 💡 Lessons Learned (Agent Workspace)
|
||||||
|
|
||||||
|
1. **Agent skills stored in `.agents/skills/`** — the canonical source for agent templates
|
||||||
|
2. **Working copies go in `03-projects/`** — keeps active projects organized alongside other work
|
||||||
|
3. **Sync when relevant** — changes to upstream agent skills should be reflected in the working copy
|
||||||
|
|
||||||
|
* * *
|
||||||
|
|
||||||
|
## 🎉 Summary
|
||||||
|
|
||||||
|
### URL Shortener
|
||||||
|
This is a **fully functional, production-ready URL shortener** that:
|
||||||
|
* ✅ Shortens URLs with 6-character codes
|
||||||
|
* ✅ Tracks every click with timestamps
|
||||||
|
* ✅ Shows live analytics with sparkline trends
|
||||||
|
* ✅ Deploys automatically on `git push`
|
||||||
|
* ✅ Runs on Cloudflare's edge (fast, free, scalable)
|
||||||
|
* ✅ Uses custom domains (`*.ai-handoff.work`)
|
||||||
|
* ✅ Has zero external dependencies (no Node.js servers, no managed DB)
|
||||||
|
|
||||||
|
**Total build time:** ~4 hours
|
||||||
|
**Monthly cost:** $0 (free tier)
|
||||||
|
**Code size:** ~600 lines (Worker + Dashboard) + ~400 lines (React app)
|
||||||
|
|
||||||
|
### Agent Workspace
|
||||||
|
* Located at `/home/user/03-projects/assistant-workspace/`
|
||||||
|
* Based on Cloudflare Agents SDK + TypeScript
|
||||||
|
* Ready for autonomous agent development
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Built with ❤️ on Cloudflare**
|
||||||
|
**Last updated:** August 12, 2026
|
||||||
@@ -0,0 +1,326 @@
|
|||||||
|
# Master Skills and Tools Documentation
|
||||||
|
|
||||||
|
This document provides a comprehensive overview of all available skills, tools, and directory structures accessible in this technical workspace.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This workspace is a sophisticated technical environment with extensive capabilities for development, testing, documentation, and system administration. It combines AI-powered development tools with traditional command-line interfaces and version control systems.
|
||||||
|
|
||||||
|
## Knowledge Base Skills
|
||||||
|
|
||||||
|
### Available Skills
|
||||||
|
|
||||||
|
#### 1. **System Processing Skills**
|
||||||
|
|
||||||
|
- **brainstorming-11** (Superpowers Brainstorming SKill)
|
||||||
|
- Purpose: Structured brainstorming and planning methodology
|
||||||
|
- Use case: Initial project planning and concept development
|
||||||
|
|
||||||
|
- **subagent-driven-development-03** (Subagent-Driven Development)
|
||||||
|
- Purpose: Execute plans by dispatching fresh implementer subagents per task with review cycles
|
||||||
|
- Use case: Complex multi-step implementation projects requiring systematic execution
|
||||||
|
|
||||||
|
- **executing-plans-01** (Executing Plans)
|
||||||
|
- Purpose: Load plan, review critically, execute all tasks, report completion
|
||||||
|
- Use case: Planning and execution of structured tasks
|
||||||
|
|
||||||
|
#### 2. **Cloudflare Development Skills**
|
||||||
|
|
||||||
|
- **workers-best-practices** (Workers Best Practices)
|
||||||
|
- Purpose: Reviews and authors Cloudflare Workers code against production best practices
|
||||||
|
- Use case: Cloudflare Workers development and code review
|
||||||
|
|
||||||
|
- **durable-objects** (Durable Objects)
|
||||||
|
- Purpose: Create and review Cloudflare Durable Objects for stateful coordination
|
||||||
|
- Use case: Chat rooms, multiplayer games, booking systems, RPC methods
|
||||||
|
|
||||||
|
- **wrangler** (Wrangler)
|
||||||
|
- Purpose: Cloudflare Workers CLI for deploying, developing, and managing Workers
|
||||||
|
- Use case: Cloudflare Workers deployment and configuration
|
||||||
|
|
||||||
|
- **building-mcp-servers-on-cloudflare-21** (Building MCP Servers)
|
||||||
|
- Purpose: Building MCP servers on Cloudflare with updated SDK knowledge
|
||||||
|
- Use case: MCP server development with Cloudflare integration
|
||||||
|
|
||||||
|
- **building-ai-agent-on-cloudflare** (Building AI Agent)
|
||||||
|
- Purpose: Building AI agents on Cloudflare with updated SDK knowledge
|
||||||
|
- Use case: AI agent development using Cloudflare Agents SDK
|
||||||
|
|
||||||
|
#### 3. **Testing and Performance Skills**
|
||||||
|
|
||||||
|
- **playwright-best-practices** (Playwright Best Practices)
|
||||||
|
- Purpose: Browser testing and screenshot capabilities
|
||||||
|
- Use case: End-to-end testing and visual regression testing
|
||||||
|
|
||||||
|
- **web-perf** (Web Performance)
|
||||||
|
- Purpose: Analyzes web performance using Chrome DevTools
|
||||||
|
- Use case: Web performance auditing and optimization
|
||||||
|
|
||||||
|
#### 4. **Business and User Experience Skills**
|
||||||
|
|
||||||
|
- **solo-dev-fiverr** (Solo Dev Fiverr)
|
||||||
|
- Purpose: Power skill for solo Node/JS/TS dev selling modern dashboard and UI work
|
||||||
|
- Use case: Client deliverables, Fiverr gigs, premium UI development
|
||||||
|
|
||||||
|
- **marketing-psychology** (Marketing Psychology)
|
||||||
|
- Purpose: Apply psychological principles to marketing
|
||||||
|
- Use case: Consumer behavior, persuasion, decision-making optimization
|
||||||
|
|
||||||
|
- **ux-content-formatte-** (UX Content Formatter)
|
||||||
|
- Purpose: Formats text, reports, and dashboard data into scannable layouts
|
||||||
|
- Use case: Content formatting, report generation, data presentation
|
||||||
|
|
||||||
|
#### 5. **Legal and Reference Skills**
|
||||||
|
|
||||||
|
- **courtlistener-api** (Courtlistener API)
|
||||||
|
- Purpose: Legal case law database with PACER data and judge profiles
|
||||||
|
- Use case: Legal research and case law analysis
|
||||||
|
|
||||||
|
#### 6. **Process Skills**
|
||||||
|
|
||||||
|
- **dispatching-parallel-agents-04** (Dispatching Parallel Agents)
|
||||||
|
- Purpose: Delegate tasks to specialized agents with isolated context
|
||||||
|
- Use case: Parallel processing and specialized task execution
|
||||||
|
|
||||||
|
- **find-skills** (Find Skills)
|
||||||
|
- Purpose: Discover and install agent skills
|
||||||
|
- Use case: Skill discovery and capability expansion
|
||||||
|
|
||||||
|
## Available Tools
|
||||||
|
|
||||||
|
### File and Directory Operations
|
||||||
|
|
||||||
|
- **list_files**: Return structured listing of files and directories
|
||||||
|
- **read_file**: Read file contents with optional line ranges and image support
|
||||||
|
- **display_file**: Open files in viewer (for visual inspection)
|
||||||
|
- **write_file**: Write text content to files with automatic directory creation
|
||||||
|
- **replace_file_content**: Find and replace exact strings in files
|
||||||
|
- **view_file**: Get file content by ID with pagination support
|
||||||
|
- **list_processes**: List all tracked background processes
|
||||||
|
|
||||||
|
### Calendar and Scheduling
|
||||||
|
|
||||||
|
- **create_calendar_event**: Create calendar events, reminders, alarms
|
||||||
|
- **search_calendar_events**: Search calendar events by text and date range
|
||||||
|
- **update_calendar_event**: Update existing calendar events
|
||||||
|
- **delete_calendar_event**: Delete calendar events permanently
|
||||||
|
|
||||||
|
### Git and Version Control
|
||||||
|
|
||||||
|
- **run_command**: Run shell commands in background with output tracking
|
||||||
|
- **send_process_input**: Write text to process stdin
|
||||||
|
- **kill_process**: Terminate processes
|
||||||
|
- **get_process_status**: Get process output and status
|
||||||
|
- **list_processes**: List all tracked background processes
|
||||||
|
|
||||||
|
### Knowledge Base Management
|
||||||
|
|
||||||
|
- **list_knowledge**: List knowledge bases, files, and notes
|
||||||
|
- **query_knowledge_files**: Semantic/vector search across knowledge files
|
||||||
|
- **view_knowledge_file**: Get content from knowledge base files
|
||||||
|
- **search_knowledge_files**: Search files by filename across knowledge bases
|
||||||
|
- **grep_knowledge_files**: Exact text search across knowledge files
|
||||||
|
- **write_note**: Create new notes with markdown content
|
||||||
|
- **view_note**: Get full note content by ID
|
||||||
|
- **search_notes**: Search saved notes by title and content
|
||||||
|
- **replace_note_content**: Update existing notes
|
||||||
|
|
||||||
|
### Project Management
|
||||||
|
|
||||||
|
- **create_tasks**: Create visible task checklists for multi-step work
|
||||||
|
- **update_task**: Mark tasks as completed/in_progress/pending/cancelled
|
||||||
|
|
||||||
|
### Documentation Generation
|
||||||
|
|
||||||
|
- **writing-plans**: Create comprehensive implementation plans with task decomposition
|
||||||
|
|
||||||
|
### Automation and Scheduling
|
||||||
|
|
||||||
|
- **create_automation**: Create scheduled automations with iCalendar RRULE
|
||||||
|
- **list_automations**: List scheduled automations
|
||||||
|
- **update_automation**: Update existing automations
|
||||||
|
- **toggle_automation**: Pause/resume automations
|
||||||
|
- **delete_automation**: Delete automations and history
|
||||||
|
|
||||||
|
### Research and Search
|
||||||
|
|
||||||
|
- **search_web**: Search the public web for current information
|
||||||
|
- **deep_research**: Multi-round research using Open WebUI web search
|
||||||
|
- **fetch_url**: Extract main text content from web pages
|
||||||
|
- **consult_council**: Orchestrates 3-stage council meetings
|
||||||
|
|
||||||
|
### Context and Library Management
|
||||||
|
|
||||||
|
- **context7_resolve-library-id**: Resolve package names to Context7-compatible library IDs
|
||||||
|
- **context7_query-docs**: Query up-to-date documentation from Context7
|
||||||
|
|
||||||
|
### Task Delegation
|
||||||
|
|
||||||
|
- **delegate_task**: Delegate focused work to parallel sub-agents
|
||||||
|
- **run_sub_agent**: Delegate tasks to sub-agents for autonomous completion
|
||||||
|
- **run_parallel_sub_agents**: Run multiple independent sub-agent tasks concurrently
|
||||||
|
|
||||||
|
### Time and Utilities
|
||||||
|
|
||||||
|
- **get_current_timestamp**: Get current Unix timestamp
|
||||||
|
- **calculate_timestamp**: Calculate timestamps for date filtering
|
||||||
|
- **timer**: Set one-shot timers
|
||||||
|
|
||||||
|
## Directory Structure
|
||||||
|
|
||||||
|
### Root and Main Directories
|
||||||
|
|
||||||
|
#### `/home/user/000-configs/` - System Configuration
|
||||||
|
|
||||||
|
- **GH_CLI_INSTRUCTIONS.md**: GitHub CLI usage instructions
|
||||||
|
- **PLAYWRIGHT_BROWSER_INSTRUCTIONS.md**: Playwright browser testing instructions
|
||||||
|
- **VITE_PREVIEW_INSTRUCTIONS.md**: Vite development server preview instructions
|
||||||
|
- **docs/**: Documentation folder with internal reference materials
|
||||||
|
- **google/**: Directory for Google-related tools or configurations
|
||||||
|
|
||||||
|
#### `/home/user/02-repos/` - Code Repositories
|
||||||
|
|
||||||
|
- **shit_in_a_vault/**: Private Obsidian vault for note-taking and knowledge management
|
||||||
|
- Contains structured markdown documentation with YAML frontmatter
|
||||||
|
- Used for project documentation and knowledge retention
|
||||||
|
|
||||||
|
- **webby/**: Separate web development project repository
|
||||||
|
- **fuckgovda_vault/**: Previously deleted locally but exists remotely Obsidian vault
|
||||||
|
|
||||||
|
#### `/home/user/02-repos/shit_in_a_vault/` - Obsidian Vault Structure
|
||||||
|
|
||||||
|
**Vault Rules and Configuration:**
|
||||||
|
|
||||||
|
- **04-models/Vault-Rules.md**: Core vault editing rules and conventions
|
||||||
|
- **04-models/model-memory.md**: Historical context and memory
|
||||||
|
- **000-configs/tools/**: Tool documentation and instructions
|
||||||
|
- **01-logs/daily/**: Automated daily notes
|
||||||
|
- **02-notes/**: Short notes, scratch work, per-topic research
|
||||||
|
- **03-archive/**: Completed or superseded materials
|
||||||
|
- **04-models/**: Rules, memory, boot prompt, model identity
|
||||||
|
- **05-handoffs/**: Model-to-model transition materials
|
||||||
|
- **06-research/**: Long-form research with citations
|
||||||
|
- **07-tasks/**: Task tracking files with metadata
|
||||||
|
- **00-assets/**: Attachments and media files
|
||||||
|
|
||||||
|
#### `/home/user/02-repos/shit_in_a_vault/04-models/` - Model Configuration
|
||||||
|
|
||||||
|
- **Vault-Rules.md**: Essential editing rules for all vault files
|
||||||
|
- **model-memory.md**: Historical context and memory
|
||||||
|
- Additional model configuration files may exist
|
||||||
|
|
||||||
|
#### `/home/user/projects/` - Development Projects
|
||||||
|
|
||||||
|
- Local project workspace for development activities
|
||||||
|
- Temporary project directories
|
||||||
|
- Experimentation and prototyping areas
|
||||||
|
|
||||||
|
## Integration and Usage Patterns
|
||||||
|
|
||||||
|
### Knowledge Management Workflow
|
||||||
|
|
||||||
|
1. **Research Phase**: Use `search_web` or `deep_research` for external information
|
||||||
|
2. **Documentation**: Store findings in `02-notes/` or `06-research/` using Obsidian vault structure
|
||||||
|
3. **Reference**: Access knowledge via `query_knowledge_files` or `search_notes`
|
||||||
|
4. **Automation**: Schedule documentation updates using `create_automation`
|
||||||
|
|
||||||
|
### Development Workflow
|
||||||
|
|
||||||
|
1. **Planning**: Use `writing-plans` for comprehensive task breakdown
|
||||||
|
2. **Implementation**: Execute tasks via `executing-plans` or `subagent-driven-development`
|
||||||
|
3. **Testing**: Use `playwright-best-practices` for browser testing
|
||||||
|
4. **Version Control**: Track changes via Git commands and `run_command`
|
||||||
|
|
||||||
|
### System Administration
|
||||||
|
|
||||||
|
1. **Process Management**: Monitor and manage background processes with `list_processes`
|
||||||
|
2. **Calendar Integration**: Schedule tasks and reminders with `create_calendar_event`
|
||||||
|
3. **Documentation Maintenance**: Keep skills and tools updated using `write_file`
|
||||||
|
|
||||||
|
## Capabilities Summary
|
||||||
|
|
||||||
|
### Core Strengths
|
||||||
|
|
||||||
|
- **Comprehensive Documentation**: Extensive tool and skill coverage
|
||||||
|
- **Multi-modal Development**: Supports both AI-powered and traditional development
|
||||||
|
- **Automation Integration**: Full scheduling and automation capabilities
|
||||||
|
- **Knowledge Management**: Sophisticated Obsidian-based documentation system
|
||||||
|
- **Testing Infrastructure**: End-to-end testing with Playwright integration
|
||||||
|
- **Cloud Integration**: Full Cloudflare Workers and Durable Objects support
|
||||||
|
- **Parallel Processing**: Multiple agent capabilities for concurrent development
|
||||||
|
|
||||||
|
### Development Spectrum
|
||||||
|
|
||||||
|
- **Simple Scripts**: Direct command-line execution
|
||||||
|
- **Complex Projects**: Multi-agent, systematic development
|
||||||
|
- **Documentation**: Comprehensive automated documentation generation
|
||||||
|
- **Testing**: Full browser automation and performance analysis
|
||||||
|
- **Deployment**: Cloud deployment with Cloudflare integration
|
||||||
|
|
||||||
|
### Collaboration Features
|
||||||
|
|
||||||
|
- **Task Delegation**: Multiple agent coordination
|
||||||
|
- **Review Cycles**: Systematic review and approval processes
|
||||||
|
- **Documentation**: Comprehensive task and project tracking
|
||||||
|
- **Knowledge Sharing**: Repository-based knowledge management
|
||||||
|
|
||||||
|
## Technical Specifications
|
||||||
|
|
||||||
|
### Environment
|
||||||
|
|
||||||
|
- **Operating System**: Linux 7.0.0-1010-aws (Ubuntu 22.04 base)
|
||||||
|
- **Shell**: bash
|
||||||
|
- **Home Directory**: `/home/user`
|
||||||
|
- **Primary Development Areas**: `/home/user`, `/home/user/projects`, `/home/user/repos`
|
||||||
|
|
||||||
|
### Authentication and Access
|
||||||
|
|
||||||
|
- **SSH Key**: Available for secure connections
|
||||||
|
- **GitHub CLI**: v2.96.0 authenticated as `stateofshit`
|
||||||
|
- **Access Levels**: Multiple access tiers for different environments
|
||||||
|
|
||||||
|
### Available Technologies
|
||||||
|
|
||||||
|
- **Languages**: Node.js, Python 3.12.13, bash
|
||||||
|
- **Frameworks**: Vite, React (preview available at `https://preview.boogerclub.com`)
|
||||||
|
- **Testing**: Playwright browser automation
|
||||||
|
- **Deployment**: Cloudflare Workers integration
|
||||||
|
- **Documentation**: Obsidian vault with markdown and YAML support
|
||||||
|
|
||||||
|
### Preview and Testing
|
||||||
|
|
||||||
|
- **Development Server**: Vite development server with preview at `https://preview.boogerclub.com`
|
||||||
|
- **Browser Testing**: Playwright screenshot and full-page testing capabilities
|
||||||
|
- **Performance Analysis**: Chrome DevTools integration for web performance
|
||||||
|
|
||||||
|
## Usage Recommendations
|
||||||
|
|
||||||
|
### For Beginners
|
||||||
|
|
||||||
|
1. Start with simple tasks using `run_command` for basic operations
|
||||||
|
2. Explore documentation in `/home/user/000-configs/`
|
||||||
|
3. Use `list_files` to understand directory structure
|
||||||
|
4. Begin with `write_note` for simple note-taking
|
||||||
|
|
||||||
|
### For Intermediate Users
|
||||||
|
|
||||||
|
1. Explore `subagent-driven-development-03` for complex projects
|
||||||
|
2. Use `search_web` for external research
|
||||||
|
3. Implement automation with `create_automation`
|
||||||
|
4. Develop comprehensive documentation with `writing-plans`
|
||||||
|
|
||||||
|
### For Advanced Users
|
||||||
|
|
||||||
|
1. Utilize parallel processing with `run_parallel_sub_agents`
|
||||||
|
2. Implement sophisticated workflows with `delegate_task`
|
||||||
|
3. Create complex Cloudflare Workers with `wrangler`
|
||||||
|
4. Build advanced automation systems with `durable-objects`
|
||||||
|
|
||||||
|
## Contact and Support
|
||||||
|
|
||||||
|
For assistance with this system:
|
||||||
|
|
||||||
|
- Refer to specific tool documentation in `/home/user/000-configs/`
|
||||||
|
- Use `search_notes` to find relevant documentation
|
||||||
|
- Consult `context7_query-docs` for library-specific information
|
||||||
|
- Contact through GitHub repositories for code-specific issu
|
||||||
@@ -0,0 +1,207 @@
|
|||||||
|
---
|
||||||
|
title: "kycourt-colab — Kentucky Appellate Court Corpus Pipeline (project notes)"
|
||||||
|
status: "active"
|
||||||
|
folder: "repo root (kycourt-colab)"
|
||||||
|
tags: [kycourt, pipeline, colab, project-notes]
|
||||||
|
created: "2026-08-12"
|
||||||
|
updated: "2026-08-12"
|
||||||
|
version: "1.1.0"
|
||||||
|
---
|
||||||
|
|
||||||
|
# kycourt-colab — Kentucky Appellate Court Corpus Pipeline
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
A single **generated Google Colab notebook** (`ky_court_pipeline.ipynb`) that
|
||||||
|
restores, discovers, downloads, extracts, and verifies the full Kentucky
|
||||||
|
appellate court document corpus (~**98.7k documents**). It is built to survive
|
||||||
|
Colab's ephemeral runtimes: it auto-resumes after disconnects, is idempotent,
|
||||||
|
and mirrors all state to Google Drive so nothing is lost between sessions.
|
||||||
|
|
||||||
|
**Remote:** https://github.com/stateofshit/kycourt-colab (private)
|
||||||
|
**Local worktree:** `/home/user/00-incoming/kycourt-colab`
|
||||||
|
|
||||||
|
## What problem it solves
|
||||||
|
|
||||||
|
A batch archival crawl of public court documents over a ~10-hour-per-session
|
||||||
|
horizon, on infrastructure (Colab) that kills your process mid-run. Requirements
|
||||||
|
that fall out of that:
|
||||||
|
|
||||||
|
- Every session must pick up exactly where the last one stopped.
|
||||||
|
- Re-running must be safe (idempotent) — no double-downloads, no data loss.
|
||||||
|
- State must live somewhere durable (Drive), because the runtime is throwaway.
|
||||||
|
- The crawl must be polite to a public API (paced, backoff, retries) and
|
||||||
|
deduplicate aggressively (content-addressed storage).
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
The notebook is **not hand-written** — its cells are inlined from real Python
|
||||||
|
modules in `notebook_builder/cells/` (one module per cell, `c00` → `c90`), and
|
||||||
|
regenerated with `notebook_builder/build.py`. This means the pipeline is
|
||||||
|
version-controlled, unit-testable Python that *just happens to be shipped as* a
|
||||||
|
Colab notebook.
|
||||||
|
|
||||||
|
Two shared abstractions carry every stage:
|
||||||
|
|
||||||
|
- **SQLite DB** (`ky_court.sqlite`) — source of truth: `documents`,
|
||||||
|
`discovery_runs`, `extraction_log`, `kv`. Opened in **WAL mode** with a
|
||||||
|
30s `busy_timeout` so a background reader can snapshot it while the writer
|
||||||
|
commits (see concurrency below).
|
||||||
|
- **`Store` abstraction** — Google Drive in Colab, local filesystem in tests;
|
||||||
|
exposes `exists / read_file / write_bytes / upload_file / list_dir`. All
|
||||||
|
stages talk to it, never to Drive directly.
|
||||||
|
|
||||||
|
**Stage order:** `restore → discover → backfill → download → extract → sync/verify`.
|
||||||
|
|
||||||
|
## The six stages (what each actually does)
|
||||||
|
|
||||||
|
### 1. Restore — `c40_restore.py`
|
||||||
|
Rehydrate a fresh/empty workspace from the newest `ky_court-WIP-resume-*.tar.gz`.
|
||||||
|
Only acts when no DB exists; skips otherwise. If a stale workspace dir is
|
||||||
|
present it is **moved aside to `-pre-restore` (renamed, never deleted)** before
|
||||||
|
extracting — the "never destroys data" guarantee. Extracts to temp, then moves
|
||||||
|
into place.
|
||||||
|
|
||||||
|
### 2. Discover — `c50_discover.py`
|
||||||
|
Enumerate the corpus by searching the court API so we know *what exists*.
|
||||||
|
Builds search shards (`YYYY-CA`, `YYYY-SC` per year; ceiling-hit shards get
|
||||||
|
sub-sharded `YYYY-CA-0..9`). Pages results with `X-CTrack-Paging-*` headers up
|
||||||
|
to a 10k ceiling; a `400` or ceiling-cross raises to force sub-sharding rather
|
||||||
|
than silently missing docs. Normalizes rows into `documents` with
|
||||||
|
`status='discovered'`, inserted idempotently via `ON CONFLICT(document_id)`.
|
||||||
|
Sweeps repeatedly until a pass adds fewer than `CONVERGENCE_DELTA` docs
|
||||||
|
(convergence). Every sweep logged to `discovery_runs`.
|
||||||
|
|
||||||
|
### 3. Backfill — `c60_backfill.py`
|
||||||
|
Deduplicate *existing* backups into the content-addressed store. Hashes every
|
||||||
|
file in `pdf_backup/` (SHA-256), writes unique content to `pdf_by_hash/<sha>.pdf`
|
||||||
|
(9,582 backup files → 3,326 unique PDFs) — "do we already have this?" becomes a
|
||||||
|
hash lookup. Normalizes DB `local_pdf` pointers. Guarded by `kv backfill_done`.
|
||||||
|
|
||||||
|
### 4. Download — `c70_download.py`
|
||||||
|
Fetch PDFs for all `discovered`/`failed` docs with `attempts < MAX_ATTEMPTS`,
|
||||||
|
in batches of 50. `api_download` does a `HEAD` (rejects non-200 and anything
|
||||||
|
> `MAX_PDF_BYTES`), then streams `GET`, hashing as it goes. Dedup wins: a hash
|
||||||
|
already in the store/manifest uploads nothing. **Retry/backoff** on transient
|
||||||
|
errors: `sleep = min(60, 5·2^attempts)` (10→20→40→60s, capped), up to 4
|
||||||
|
attempts. Permanent errors (400/401/403/404/**413**) fail immediately, no
|
||||||
|
backoff. Writes to `tmp/<id>.part`, promotes to store, paces itself, and pushes
|
||||||
|
a heartbeat to `status.json` every 100 attempts.
|
||||||
|
|
||||||
|
### 5. Extract — `c80_extract.py`
|
||||||
|
Turn downloaded PDFs into searchable text with `pypdf`. Batches of 500 with 8
|
||||||
|
parallel worker threads. A PDF with no text layer is a valid `"empty"` result,
|
||||||
|
not an error. Text written to `text/<sha>.txt`; `extracted_chars` recorded.
|
||||||
|
Heuristic flags likely-scanned (image-only) PDFs when `chars/pages < 20`.
|
||||||
|
|
||||||
|
### 6. Sync/Verify — `c90_verify.py`
|
||||||
|
Prove nothing is corrupted and push an authoritative snapshot + summaries to
|
||||||
|
Drive. Runs `PRAGMA integrity_check`; counts totals and the **remaining
|
||||||
|
queues** (download + extract eligibility queries mirror the stage code exactly,
|
||||||
|
so `done` is trustworthy). Exports `document_inventory.csv` +
|
||||||
|
`failed_inventory.csv`. `checkpoint()` makes a consistent DB snapshot via
|
||||||
|
`src.backup(dst)` (SQLite online backup API) and uploads it as the new
|
||||||
|
`ky_court.sqlite`. Updates `status.json` with `done = (queues empty)`.
|
||||||
|
Prints `PASS`/`FAIL`.
|
||||||
|
|
||||||
|
## Concurrency (the interesting part)
|
||||||
|
|
||||||
|
`checkpoint()` uses SQLite's **online backup API** (`src.backup(dst)`) from a
|
||||||
|
**separate connection** — not `copytree`, not a WAL checkpoint. It copies the
|
||||||
|
DB page-by-page and re-copies any page the writer modified during the pass,
|
||||||
|
looping until a clean pass completes. Combined with **WAL mode** (readers don't
|
||||||
|
block the writer) and **`busy_timeout=30000`**, this yields a
|
||||||
|
transactionally-consistent snapshot even while Download is committing every doc.
|
||||||
|
A naive file copy would produce a torn/corrupt snapshot and drop WAL-committed
|
||||||
|
data.
|
||||||
|
|
||||||
|
## Key design decisions
|
||||||
|
|
||||||
|
- **Idempotent + resume-safe everywhere**: skip what's done, never auto-delete
|
||||||
|
on Drive.
|
||||||
|
- **Named permanent-failure set** (`PERMANENT_STATUSES` + `_is_permanent()`)
|
||||||
|
distinguishes never-succeed (401/403/404/413) from transient (5xx/conn/429/408).
|
||||||
|
- **Content-addressed store** (`pdf_by_hash/<sha>.pdf`) makes dedup a lookup.
|
||||||
|
- **Config via one `CFG` dict** with env overrides (`KY_BUDGET`, `KY_DRY`,
|
||||||
|
`KY_PHASES`, `KY_RETRY_HARD`, `KY_LIVE`).
|
||||||
|
|
||||||
|
## Config knobs (top cell `CFG`)
|
||||||
|
|
||||||
|
| Key | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `BUDGET_SECONDS` | 10h | max wall-clock per session (`KY_BUDGET`) |
|
||||||
|
| `SHUTDOWN_SECONDS` | 15 min | stop early so the session finishes cleanly |
|
||||||
|
| `PACING_SEARCH` / `PACING_DL` | (0.3,0.7)/(0.3,0.5) | random sleep between API calls |
|
||||||
|
| `MAX_ATTEMPTS` | 4 | download retries before `failed` |
|
||||||
|
| `MAX_PDF_BYTES` | 200 MiB | size cap; oversized (413) fails permanently, no retry/backoff |
|
||||||
|
| `MAX_EXTRACT_ATTEMPTS` | 3 | extraction retries before skip |
|
||||||
|
| `RETRY_HARD` | 0 (`KY_RETRY_HARD`) | on resume, reset `failed` → `discovered` |
|
||||||
|
| `PHASES` | discover,backfill,download,extract | stages to run (`KY_PHASES`) |
|
||||||
|
| `DRY_RUN` | 0 (`KY_DRY`) | 1 = fully offline, no network |
|
||||||
|
|
||||||
|
## File layout
|
||||||
|
|
||||||
|
```
|
||||||
|
ky_court_pipeline.ipynb the deliverable (generated)
|
||||||
|
notebook_builder/
|
||||||
|
build.py regenerate the notebook from cells
|
||||||
|
cells/ c00_config … c90_verify + drive_store.py
|
||||||
|
tests/ test_pipeline_local.py + conftest.py (offline; live gated by KY_LIVE)
|
||||||
|
scripts/ smoke_local.py full offline pipeline over the real DB
|
||||||
|
RUNBOOK.md operational runbook (first run, resuming, failure modes)
|
||||||
|
docs/superpowers/ specs/ + plans/ design spec + implementation plan
|
||||||
|
PROJECT_NOTES.md this file
|
||||||
|
.gitignore ignores .venv, extracted/, *.sqlite, __pycache__,
|
||||||
|
.worktrees/, .superpowers/ (NOT the WIP tars)
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Data is NOT in the repo.** `extracted/`, `*.tar.gz` resume/WIP archives, and
|
||||||
|
> `sanitize-work/` live only in Drive / locally. Gitignore does not cover the
|
||||||
|
> WIP `.tar.gz` files, so never `git add -A` from the repo root.
|
||||||
|
|
||||||
|
## Running / testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
|
||||||
|
.venv/bin/pytest tests -q # 20 offline tests (live ones skipped)
|
||||||
|
.venv/bin/python scripts/smoke_local.py # full pipeline over the real 98,724-row DB
|
||||||
|
.venv/bin/python notebook_builder/build.py # regenerate the notebook
|
||||||
|
```
|
||||||
|
|
||||||
|
Live API tests are gated behind `KY_LIVE=1` and share a ≤10-request budget.
|
||||||
|
Suite result at v1.2: **20 passed, 3 skipped**.
|
||||||
|
|
||||||
|
## Failure modes (from RUNBOOK)
|
||||||
|
|
||||||
|
- **Colab disconnect mid-run** → re-run; `recover()` heals stale rows, drops
|
||||||
|
`.part` files.
|
||||||
|
- **Drive quota exceeded** → checkpoint push fails; run continues, store lags.
|
||||||
|
- **5xx/connection storms** → backoff `min(60, 5·2^attempts)` up to `MAX_ATTEMPTS`.
|
||||||
|
- **Search shard hits 10k ceiling** → planner sub-shards next run.
|
||||||
|
- **`integrity_check` not "ok"** → DB corrupt; restore from last resume tar.
|
||||||
|
|
||||||
|
## Versioning / releases
|
||||||
|
|
||||||
|
Immutable milestone tags (never force-move a published tag):
|
||||||
|
|
||||||
|
```
|
||||||
|
pipeline-v1 33a6ec1 original pipeline milestone
|
||||||
|
v1.1 ae77344 +README
|
||||||
|
v1.2 ec248fe +413 oversize permanent-failure fix (MAX_PDF_BYTES knob)
|
||||||
|
```
|
||||||
|
|
||||||
|
`v1.2` also has a **GitHub release** with changelog:
|
||||||
|
https://github.com/stateofshit/kycourt-colab/releases/tag/v1.2
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Project complete and shipped. Suite green offline. The pipeline is ready to run in Colab (restore from the resume tar → Run all).
|
||||||
|
|
||||||
|
### Roadmap / Future enhancements
|
||||||
|
- **Parallel extract workers** — tune the 8-worker batch size based on observed Colab runtime limits.
|
||||||
|
- **Drive quota monitoring** — automated alert when `status.json` reports approaching quota, with graceful degradation to skip upload.
|
||||||
|
- **CLI entry point** — `kycourt-pipeline run --phase discover` to run individual stages outside Colab.
|
||||||
|
- **Query layer** — lightweight read-only API over the SQLite DB for inspecting discovered/extracted docs without re-running the pipeline.
|
||||||
|
- **Vault migration** — if/when the repo is folded into the vault, adapt `folder` frontmatter and consolidate `.tar.gz` WIP archives under the vault's resume format. Remote backup, README, tags, and v1.2 release all
|
||||||
|
in place on GitHub. Suite green offline. The pipeline is ready to run in Colab
|
||||||
|
(restore from the resume tar → Run all).
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# ky-court-colab
|
||||||
|
|
||||||
|
A single generated Google Colab notebook that **restores, discovers, downloads,
|
||||||
|
extracts, and verifies** the full Kentucky appellate court document corpus
|
||||||
|
(~98.7k documents). Built to survive Colab disconnects — it auto-resumes, is
|
||||||
|
idempotent, and mirrors all state to Google Drive so nothing is lost between
|
||||||
|
sessions.
|
||||||
|
|
||||||
|
## What it does
|
||||||
|
|
||||||
|
`ky_court_pipeline.ipynb` walks the corpus through six stages:
|
||||||
|
|
||||||
|
```
|
||||||
|
restore → discover → backfill → download → extract → sync/verify
|
||||||
|
```
|
||||||
|
|
||||||
|
Every stage reads/writes one SQLite DB and a `Store` abstraction (Google Drive
|
||||||
|
in Colab, local filesystem in tests). Any stage stops gracefully at the session
|
||||||
|
time budget; re-running resumes exactly where it left off.
|
||||||
|
|
||||||
|
## Highlights
|
||||||
|
|
||||||
|
- **Resumable** — `recover()` heals stale state; each stage skips what's done.
|
||||||
|
- **Idempotent** — safe to re-run as often as you like.
|
||||||
|
- **Drive-backed** — SQLite DB, deduped PDFs, extracted text, exports, and a
|
||||||
|
live `status.json` heartbeat all live under `MyDrive/ky_court-WIP/`.
|
||||||
|
- **Never auto-deletes on Drive** — original backups are retained.
|
||||||
|
- **Polite + fault-tolerant** — paced API calls, exponential backoff, capped
|
||||||
|
retries, and a `DRY_RUN` mode for fully offline exercise.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
| Path | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `ky_court_pipeline.ipynb` | the deliverable (generated notebook) |
|
||||||
|
| `notebook_builder/` | sources: `build.py` + one module per cell (`c00_config` … `c90_verify`) |
|
||||||
|
| `tests/` | offline pytest suite (18 tests; live ones gated behind `KY_LIVE=1`) |
|
||||||
|
| `scripts/smoke_local.py` | full offline pipeline over the real DB |
|
||||||
|
| `RUNBOOK.md` | full operational runbook (first run, resuming, failure modes, config knobs) |
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# local test/smoke
|
||||||
|
python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
|
||||||
|
.venv/bin/pytest tests -q
|
||||||
|
.venv/bin/python scripts/smoke_local.py
|
||||||
|
.venv/bin/python notebook_builder/build.py # regenerate the notebook
|
||||||
|
|
||||||
|
# on Google Colab
|
||||||
|
# upload ky_court-WIP-resume-*.tar.gz to MyDrive/ (if Drive is empty),
|
||||||
|
# open ky_court_pipeline.ipynb, Runtime > Run all.
|
||||||
|
```
|
||||||
|
|
||||||
|
See **`RUNBOOK.md`** for the full runbook — first run, resuming, what "done"
|
||||||
|
looks like, failure modes, and every config knob.
|
||||||
|
|
||||||
|
> **Note:** input data (extracted archives, resume tarballs, `_WIP` backups)
|
||||||
|
> is deliberately **not** in this repo — it lives in Google Drive / locally.
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
# RUNBOOK — ky_court_pipeline.ipynb
|
||||||
|
|
||||||
|
One notebook to restore, discover, download, extract, sync, and verify the full
|
||||||
|
Kentucky appellate court document corpus (~98.7k documents). It is designed to
|
||||||
|
run in Google Colab, auto-resume after disconnects, and mirror all state up to a
|
||||||
|
Drive folder so nothing is lost across sessions.
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
A single generated notebook (`ky_court_pipeline.ipynb`) whose cells are inlined
|
||||||
|
from real Python modules in `notebook_builder/cells/`. The notebook is rebuilt
|
||||||
|
with `notebook_builder/build.py`. All stages read/write the same SQLite DB and a
|
||||||
|
`Store` abstraction that is Google Drive in Colab and the local filesystem in
|
||||||
|
tests/smoke.
|
||||||
|
|
||||||
|
Stage order: **restore → discover → backfill → download → extract → sync/verify**.
|
||||||
|
Any stage stops gracefully at the session budget; re-running resumes where it
|
||||||
|
left off.
|
||||||
|
|
||||||
|
## Files on Drive (`MyDrive/ky_court-WIP/`)
|
||||||
|
|
||||||
|
| Path | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `ky_court.sqlite` | source of truth (documents, discovery_runs, kv, extraction_log) |
|
||||||
|
| `pdf_by_hash/<sha256>.pdf` | deduped PDFs keyed by content hash (3,326 unique from 9,582 backup files) |
|
||||||
|
| `pdf_backup/<case>_<n>.pdf` | original named backups — **retained, never deleted** |
|
||||||
|
| `text/<sha256>.txt` | extracted text per PDF |
|
||||||
|
| `exports/file_manifest.csv` | sha256 → size manifest |
|
||||||
|
| `exports/document_inventory.csv` / `failed_inventory.csv` | exported summaries from verify |
|
||||||
|
| `status.json` | live heartbeat + phase/counts (the "is it alive" check) |
|
||||||
|
| `ky_court-WIP-resume-*.tar.gz` | (MyDrive root) resume tarball used to rehydrate a fresh runtime |
|
||||||
|
|
||||||
|
The store root is `MyDrive/ky_court-WIP` (Colab) or `<WORKDIR>/ky_court-WIP` (local).
|
||||||
|
|
||||||
|
## First run
|
||||||
|
|
||||||
|
1. If Drive is empty (no `ky_court-WIP`), upload `ky_court-WIP-resume-YYYY-MM-DD.tar.gz`
|
||||||
|
to `MyDrive/`. Stage **restore** will extract it.
|
||||||
|
2. Open `ky_court_pipeline.ipynb` in Colab.
|
||||||
|
3. Edit the first code cell (`CFG`) if needed (budget, phases — see Config knobs).
|
||||||
|
4. **Runtime > Run all**. A session mounts Drive, opens the DB (pulling it from
|
||||||
|
the store if absent), heals stale state, then runs the enabled phases.
|
||||||
|
|
||||||
|
Expected session timeline (typical):
|
||||||
|
- setup: ~1 min (Drive mount, dirs, recover)
|
||||||
|
- discover: 10–30 min (74 shards, paged search, paced)
|
||||||
|
- backfill: ~20 min (hash `pdf_backup` → dedupe into `pdf_by_hash`)
|
||||||
|
- download: ~10 h per session ≈ 6–12k documents (budget-capped)
|
||||||
|
- extract: rides along between batches (thread-pooled PDF text extraction)
|
||||||
|
- verify: ~5 min (integrity check, inventory exports, status push)
|
||||||
|
|
||||||
|
## Resuming
|
||||||
|
|
||||||
|
Just re-run the notebook. `recover()` closes any stale `running` discovery runs,
|
||||||
|
drops leftover `.part` files, and each stage skips what's already done:
|
||||||
|
- backfill: guarded by `kv backfill_done`
|
||||||
|
- download: only `discovered`/`failed` docs with `attempts < MAX_ATTEMPTS`
|
||||||
|
- extract: only `downloaded` docs with no extracted text yet
|
||||||
|
|
||||||
|
Idempotent by design — safe to rerun as often as you like.
|
||||||
|
|
||||||
|
## What "done" looks like
|
||||||
|
|
||||||
|
Verify prints `PASS {checks}` and, when every queue is empty:
|
||||||
|
- notebook verify output includes `done: True`
|
||||||
|
- `status.json` on Drive contains `"done": true`
|
||||||
|
|
||||||
|
A session that merely hit its budget is **not** "done" — it stops early with
|
||||||
|
`status.json` showing the phase and remaining queues. That is normal mid-flight
|
||||||
|
state, not an error.
|
||||||
|
|
||||||
|
## Failure modes
|
||||||
|
|
||||||
|
- **Colab disconnect mid-run** → just re-run. `recover()` heals stale rows;
|
||||||
|
`.part` files are dropped; partial downloads retry from the queue.
|
||||||
|
- **Drive quota exceeded** → verify/checkpoint push fails; the run continues
|
||||||
|
locally in the session but the store lags. Check Drive quota, free space, re-run.
|
||||||
|
- **5xx / connection storms** → `api_download` returns an error tuple and
|
||||||
|
`stage_download` backs off `min(60, 5*2^attempts)` seconds, retrying up to
|
||||||
|
`MAX_ATTEMPTS` before marking the doc `failed`.
|
||||||
|
- **Search shard hits the 10k ceiling** → `api_search_pages` raises; the planner
|
||||||
|
sub-shards that seed (`YYYY-CA-N`) on the next run.
|
||||||
|
- **`integrity` check not "ok"** → verify prints `FAIL`; the DB is corrupted and
|
||||||
|
should be restored from the last resume tar before trusting anything.
|
||||||
|
|
||||||
|
## Config knobs (top code cell `CFG`)
|
||||||
|
|
||||||
|
| Key | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `BUDGET_SECONDS` | 10 * 3600 | max wall-clock per session (env `KY_BUDGET`) |
|
||||||
|
| `SHUTDOWN_SECONDS` | 900 | stop stages this long before the deadline so the session can finish cleanly |
|
||||||
|
| `PACING_SEARCH` / `PACING_DL` | (0.3,0.7) / (0.3,0.5) | random sleep range between API calls (polite throttling) |
|
||||||
|
| `MAX_ATTEMPTS` | 4 | download retries before `failed` |
|
||||||
|
| `MAX_PDF_BYTES` | 200 MiB | size cap; oversized (HTTP 413) fails permanently, no retry/backoff |
|
||||||
|
| `MAX_EXTRACT_ATTEMPTS` | 3 | PDF-extraction retries before skip |
|
||||||
|
| `RETRY_HARD` | 0 (env `KY_RETRY_HARD`) | on resume, reset `failed` docs back to `discovered` |
|
||||||
|
| `PHASES` | discover,backfill,download,extract | comma-separated phases to run (env `KY_PHASES`) |
|
||||||
|
| `DRY_RUN` | 0 (env `KY_DRY`) | 1 = fully offline: discovery/download skip all network |
|
||||||
|
|
||||||
|
## Safety notes
|
||||||
|
|
||||||
|
- The pipeline **never deletes on Drive automatically**. `pdf_backup` is retained;
|
||||||
|
pre-restore move-aside only renames (`-pre-restore`), never deletes.
|
||||||
|
- Set `DRY_RUN=1` (`KY_DRY=1`) to exercise the whole flow offline with zero
|
||||||
|
network calls — the smoke script (`scripts/smoke_local.py`) does exactly this.
|
||||||
|
- Live API tests are gated behind `KY_LIVE=1` and share a ≤10-request budget;
|
||||||
|
keep them there so the ordinary test suite stays offline and fast.
|
||||||
|
|
||||||
|
## Local / tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m venv .venv && .venv/bin/pip install -r <pinned deps>
|
||||||
|
.venv/bin/pytest tests -q # 18 offline tests (live ones skipped)
|
||||||
|
.venv/bin/python scripts/smoke_local.py # full pipeline over the real 98,724-row DB
|
||||||
|
.venv/bin/python notebook_builder/build.py # regenerate the notebook
|
||||||
|
```
|
||||||
@@ -1,3 +1,7 @@
|
|||||||
|
---
|
||||||
|
created: 2026-08-06T15:28:00
|
||||||
|
updated: 2026-08-12T15:28:00
|
||||||
|
---
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -161,3 +165,19 @@ DRIVE_API_KEY=AIzaSyAB5mry4SGDb1bmNGLGZ4bmDPXAUai6HPs
|
|||||||
## pinecone
|
## pinecone
|
||||||
KEY=pcsk_Ats6a_7tukrjtHVJskhZDG8zVDBjG5X4PNPrVECaGVRU3B137x432U62SfbV2495Psknd
|
KEY=pcsk_Ats6a_7tukrjtHVJskhZDG8zVDBjG5X4PNPrVECaGVRU3B137x432U62SfbV2495Psknd
|
||||||
```
|
```
|
||||||
|
```
|
||||||
|
# thestateofshit@gmail.com
|
||||||
|
render_api_key=rnd_PyfjhLC3KKG43qazh5TbA9aJsO62
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# medic8dcloud@gmail.com google ai studio
|
||||||
|
key=AIzaSyDZfMTQ0ayi3phYTKwdpI7uPPtDYfdvgZk
|
||||||
|
key=8RN6IOh97RbGeoixINNWp7QRtt-_LBKmJD-xuhugZ1XurDZA
|
||||||
|
## for drive,docs,sheets,gmail,calander
|
||||||
|
api_key=AIzaSyASs9IQteSVtV61fkUAR9mu9JkB0sMw1Vw
|
||||||
|
---
|
||||||
|
# medic8cloud qdrant
|
||||||
|
apikey=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3MiOiJtIiwic3ViamVjdCI6ImFwaS1rZXk6NjgzYTc4ZGQtNmY4Mi00ZWMxLTg0ODYtNWU3YmY0NGY0OGRhIn0.WiO_I07EuUSDQVI1w2VzxBkXTPyIEUfkWtc_DU_gmQk
|
||||||
|
enpoint=https://0ff33b75-b828-417f-8c60-e219d442ee2d.us-east-1-1.aws.cloud.qdrant.io
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user