diff --git a/000-configs/README.md b/000-configs/README.md index 5d46d54..5f09ac4 100644 --- a/000-configs/README.md +++ b/000-configs/README.md @@ -21,6 +21,7 @@ Server, model, and workspace configuration docs. Mirror of `/home/user/000-confi | [[skills/]] | Skill definitions | | [[modes/]] | Model mode presets (creative, research, code, etc.) — incl. `workplace-modes.md` | | [[workspace/]] | Workspace layouts and setup | +| [[notebooks/]] | Jupyter / Colab notebooks (runnable .ipynb artifacts) | | [[settings/]] | Open WebUI / server settings | ## Migration Note diff --git a/000-configs/notebooks/README.md b/000-configs/notebooks/README.md new file mode 100644 index 0000000..f7eac86 --- /dev/null +++ b/000-configs/notebooks/README.md @@ -0,0 +1,32 @@ +--- +title: "Notebooks" +status: "active" +folder: "000-configs/notebooks" +tags: [readme, notebook, code] +created: "2026-07-30" +updated: "2026-07-30" +version: "1.0.0" +--- + +# Notebooks + +Executable Jupyter / Colab notebooks that accompany research projects or workspace workflows. These are *runnable* artifacts (.ipynb), not docs — the companion writeup lives in `06-research/` or as a task in `07-tasks/`. + +## How to use + +1. Download the `.ipynb` from this folder (or `git pull` on your PC and open in Obsidian / Jupyter / Colab). +2. Open in **Google Colab** for the Cloud-Drive-backed workflows (most notebooks here assume Colab free tier + Drive mount). +3. Update the task file in `07-tasks/` when you re-run or modify. + +## Current notebooks + +| Notebook | Purpose | Companion task | +|---|---|---| +| [[docx_to_markdown_openwebui.ipynb]] | DOCX + MD → clean Markdown for Open WebUI knowledge bases (mammoth + python-docx fallback + fuzzy dedup + URL extraction) | [[07-tasks/docx-to-markdown-converter-notebook]] | + +## When adding a new notebook + +1. Drop the `.ipynb` here. +2. Add a row to the table above with a link. +3. Create or update a task in `07-tasks/.md` describing what it does and how to run it. +4. Commit with `notebook: ` per the repo commit-message convention. diff --git a/000-configs/notebooks/docx_to_markdown_openwebui.ipynb b/000-configs/notebooks/docx_to_markdown_openwebui.ipynb new file mode 100644 index 0000000..ffda5cc --- /dev/null +++ b/000-configs/notebooks/docx_to_markdown_openwebui.ipynb @@ -0,0 +1,1335 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "782d203a", + "metadata": {}, + "source": [ + "# DOCX & MD → Markdown for Open WebUI Knowledge Bases\n", + "\n", + "Convert `.docx` and `.md` files into clean Markdown optimized for embedding into **Open WebUI** knowledge bases.\n", + "\n", + "**Why this matters for Open WebUI:** Open WebUI supports *Markdown Header Splitting* (H1–H6) when chunking documents for RAG — it splits on headers first, then merges tiny fragments via the *Chunk Min Size Target* setting. So a converter that preserves clean, well-nested heading hierarchy produces far better retrieval chunks than flat text. [Open WebUI RAG docs](https://docs.openwebui.com/features/chat-conversations/rag/)\n", + "\n", + "This notebook is designed for the **Colab free tier**:\n", + "- No GPU needed (CPU only).\n", + "- Lightweight pip installs (`mammoth`, `python-docx`, `markdownify`, `requests`, `beautifulsoup4`).\n", + "- Inputs and outputs are mirrored to Google Drive so the work survives Colab wipes.\n", + "\n", + "**What this notebook does:**\n", + "1. Mounts your Google Drive and uses the persistent folder `/drive/MyDrive/to_convert/` to track every file you’ve ever processed.\n", + "2. Accepts `.docx` and `.md` inputs (uploaded directly, or auto-extracted from a `.tar` archive in Drive).\n", + "3. **Dedupes by content hash** (SHA-256 of the original bytes) — if you re-run a file with a slightly different filename, the manifest catches it and skips. If the bytes actually differ, it reconverts the new content.\n", + "4. Converts via the same mammoth pipeline + python-docx fallback as the previous notebook (all heading-structure cleanup preserved).\n", + "5. Cleans up raw `.md` inputs through the same post-processing pipeline (strip cite markers, collapse HR, promote bold to headings, collapse extra H1s).\n", + "6. Extracts every `http(s)://` URL from each converted file and appends to `sources.csv` (paired with the source doc’s SHA + stem).\n", + "7. Optionally bulk-downloads those URLs (PDFs first, then HTML), tracked in `downloads.csv`.\n", + "\n", + "**Pipeline (per file):**\n", + "- DOCX: `mammoth(html) → clean_markdown → strip_masthead → strip_cite_markers → collapse_hr → promote_structure → collapse_extra_h1 → conditional synthetic H1 prepend`\n", + "- MD: `clean_markdown → strip_masthead → strip_cite_markers → collapse_hr → promote_structure → collapse_extra_h1 → conditional synthetic H1 prepend` (no mammoth pass)\n", + "\n", + "> ⚠️ Open WebUI's Temporary Chat mode disables backend parsing for complex DOCX, so it's better to upload **pre-converted Markdown** like the output of this notebook rather than raw DOCX.\n", + "\n", + "> **Auto heading promotion:** For Gemini-Notebook-style DOCX files that use bold paragraphs (not Word heading styles) for section titles, the converter promotes those bold lines into real `## / ###` Markdown headings so Open WebUI's Markdown Header Splitter actually has structure to chunk on. It also strips `[cite: N]` Gemini citation artifacts and the redundant in-document title (the filename stem becomes the single `#` H1).\n" + ] + }, + { + "cell_type": "markdown", + "id": "c00618e1", + "metadata": {}, + "source": [ + "## 0. Tips before you run\n", + "\n", + "- **Heading styles matter.** Mammoth relies on Word *paragraph styles* (e.g. `Heading 1`, `Heading 2`) to produce `#`, `##`. If your document was authored with manual bold/large fonts instead of styles, headings may come out as plain paragraphs. You can remap styles via the `STYLE_MAP` cell below.\n", + "- **One source Word style → one Markdown heading level.** Keep it linear (Heading 1 → `#`, Heading 2 → `##`, …). Open WebUI's header splitter rewards clean hierarchies.\n", + "- **Chunk size tuning happens in Open WebUI**, not here. After upload, set `Chunk Size` (~1000–2000 chars) and `Chunk Min Size Target` (e.g. ~half the chunk size) under **Admin → Tools → Documents**. The docs note this can cut chunk counts by >90% while improving retrieval quality.- **If your DOCX uses real Word heading styles** (Heading 1/2/...),\n", + " Mammoth emits them as `# / ## / ...` directly and the promotion step is\n", + " a no-op for those lines. The promotion only fires on *bold-only* paragraphs,\n", + " which is how the Gemini Notebook exports are structured.\n" + ] + }, + { + "cell_type": "markdown", + "id": "390e295a", + "metadata": {}, + "source": [ + "## 0. Install dependencies\\n\\nLightweight and CPU-only." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ca2584b1", + "metadata": {}, + "outputs": [], + "source": [ + "# Run once per session. ~5–10 seconds on free tier.\n", + "%pip -q install mammoth==1.9.1 markdownify python-docx==1.1.2 requests beautifulsoup4" + ] + }, + { + "cell_type": "markdown", + "id": "75bc63c5", + "metadata": {}, + "source": [ + "## 1. Mount Google Drive and create the folder tree\n", + "\n", + "You create the empty folder `to_convert/` on your Google Drive **once** (at `/My Drive/to_convert/`). The notebook then creates the rest of the tree on first run:\n", + "\n", + "```\n", + "/drive/MyDrive/to_convert/\n", + "├── manifest.csv ← SHA-256 dedup history (append-only)\n", + "├── archive/ ← original input bytes, durable across Colab wipes\n", + "├── markdown_output/ ← converted .md files\n", + "├── sources.csv ← doc → URLs (append-only)\n", + "└── downloads/ ← bulk URL fetch output\n", + " ├── pdf/ html/ other/\n", + " └── downloads.csv ← URL → saved file (append-only)\n", + "```\n", + "\n", + "The manifest is the source of truth for \"have I run this file before?\" — it stores the SHA-256 of the *original input bytes*, so renaming a file does NOT cause a reconvert." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0bd89046", + "metadata": {}, + "outputs": [], + "source": [ + "import os, shutil, csv, hashlib, re, datetime, time\n", + "from pathlib import Path\n", + "from google.colab import drive\n", + "\n", + "drive.mount('/content/drive')\n", + "\n", + "DRIVE_ROOT = Path('/content/drive/MyDrive/to_convert')\n", + "ARCHIVE_DIR = DRIVE_ROOT / 'archive'\n", + "MD_OUTPUT_DIR = DRIVE_ROOT / 'markdown_output'\n", + "DOWNLOADS_DIR = DRIVE_ROOT / 'downloads'\n", + "DOWNLOADS_CSV = DOWNLOADS_DIR / 'downloads.csv'\n", + "MANIFEST_PATH = DRIVE_ROOT / 'manifest.csv'\n", + "SOURCES_PATH = DRIVE_ROOT / 'sources.csv'\n", + "\n", + "# Create the folder tree (mkdir -p). User just creates /to_convert/ once on Drive.\n", + "for p in (DRIVE_ROOT, ARCHIVE_DIR, MD_OUTPUT_DIR,\n", + " DOWNLOADS_DIR, DOWNLOADS_DIR / 'pdf',\n", + " DOWNLOADS_DIR / 'html', DOWNLOADS_DIR / 'other'):\n", + " p.mkdir(parents=True, exist_ok=True)\n", + "\n", + "# Local scratch input dir (wiped each run so uploads don't accumulate).\n", + "INPUT_DIR = Path('/content/docx_input')\n", + "if INPUT_DIR.exists():\n", + " shutil.rmtree(INPUT_DIR)\n", + "INPUT_DIR.mkdir(parents=True, exist_ok=True)\n", + "\n", + "print(f'Drive root: {DRIVE_ROOT}')\n", + "print(f'Local input: {INPUT_DIR}')\n", + "print(f'MD output: {MD_OUTPUT_DIR}')" + ] + }, + { + "cell_type": "markdown", + "id": "27740394", + "metadata": {}, + "source": [ + "## 2. Provide `.docx` and/or `.md` inputs\n", + "\n", + "Two ways to get inputs into the notebook:\n", + "\n", + "1. **Auto-extract from a `.tar`/`.tgz` archive in Drive.** If you keep an archive at `/drive/MyDrive/zzz-new_shit/`, the cell below will copy and extract it automatically. Edit `DRIVE_ARCHIVE_SOURCE_DIR` if your archive lives elsewhere.\n", + "2. **Upload via the Colab file picker.** If no archive is found (or you comment out the archive path), the cell falls back to `google.colab.files.upload()` so you can hand-pick files.\n", + "\n", + "The cell accepts both `.docx` and `.md` extensions and merges them into one `input_files` list." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f12edf78", + "metadata": {}, + "outputs": [], + "source": [ + "import os, shutil, tarfile\n", + "from pathlib import Path\n", + "from google.colab import files as colab_files\n", + "\n", + "# Edit this to point at a Drive folder that holds your archive, or comment it out to use the upload fallback.\n", + "DRIVE_ARCHIVE_SOURCE_DIR = Path('/content/drive/MyDrive/zzz-new_shit')\n", + "\n", + "tar_archive_path = None\n", + "if DRIVE_ARCHIVE_SOURCE_DIR.exists():\n", + " for item in DRIVE_ARCHIVE_SOURCE_DIR.glob('*.tar'):\n", + " if item.is_file():\n", + " tar_archive_path = item\n", + " break\n", + " if not tar_archive_path:\n", + " for item in DRIVE_ARCHIVE_SOURCE_DIR.glob('*.tgz'):\n", + " if item.is_file():\n", + " tar_archive_path = item\n", + " break\n", + "\n", + "if tar_archive_path:\n", + " print(f\"Found archive: {tar_archive_path.name}\")\n", + " temp_archive_path = Path('/content') / tar_archive_path.name\n", + " print(f\"Copying '{tar_archive_path}' to '{temp_archive_path}'...\")\n", + " shutil.copy2(tar_archive_path, temp_archive_path)\n", + " print(\"Copy complete.\")\n", + " print(f\"Extracting '{temp_archive_path}' to '{INPUT_DIR}'...\")\n", + " with tarfile.open(temp_archive_path, \"r\") as tar:\n", + " tar.extractall(path=INPUT_DIR, filter=\"data\")\n", + " print(\"Extraction complete.\")\n", + " temp_archive_path.unlink()\n", + "else:\n", + " print(f\"No archive in {DRIVE_ARCHIVE_SOURCE_DIR} — using Colab upload picker.\")\n", + " print(\"Pick .docx and/or .md files in the dialog that appears.\")\n", + " uploaded = colab_files.upload()\n", + " if not uploaded:\n", + " raise RuntimeError(\"No files uploaded.\")\n", + " for name, data in uploaded.items():\n", + " (INPUT_DIR / name).write_bytes(data)\n", + " print(f\"Uploaded {len(uploaded)} file(s) into {INPUT_DIR}.\")\n", + "\n", + "# Discover inputs (recursively, in case the tar extracted into subdirs).\n", + "input_files = sorted([*INPUT_DIR.glob('**/*.docx'), *INPUT_DIR.glob('**/*.md')])\n", + "n_docx = sum(1 for f in input_files if f.suffix.lower() == '.docx')\n", + "n_md = sum(1 for f in input_files if f.suffix.lower() == '.md')\n", + "\n", + "print(f\"\\nFound {len(input_files)} input file(s): {n_docx} .docx, {n_md} .md\")\n", + "for f in input_files:\n", + " print(' •', f.relative_to(INPUT_DIR))\n", + "\n", + "# Snapshot the original upload list so the pre-flight dedup cell (5b) can be\n", + "# re-run safely without the list shrinking on each pass (idempotent).\n", + "uploaded_files_master = list(input_files)" + ] + }, + { + "cell_type": "markdown", + "id": "7f526dc6", + "metadata": {}, + "source": [ + "## 3. Config & Mammoth style map\n", + "\n", + "Configure dedup/archival/download flags here. The style map below is preserved from the previous notebook — it maps Word `Title`/`Heading N` styles to HTML `` tags so mammoth emits real headings." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "584e5c0e", + "metadata": {}, + "outputs": [], + "source": [ + "# === Config ===\n", + "SKIP_ALREADY_SEEN = True # if True, skip files whose sha256 is already in manifest.csv\n", + "ARCHIVE_INPUTS = True # copy each input's bytes to archive/ for future hash matching\n", + "DOWNLOAD_SOURCES = False # only the explicit downloader cell flips this on\n", + "RATE_LIMIT_SEC = 1.0 # politeness delay between URL fetches\n", + "\n", + "# Pre-flight near-duplicate detection (cell 5b). Two files count as 'the same doc'\n", + "# when their overlapping-fragment ratio is >= this threshold. 0.97 means '~95-97%'\n", + "# chunk overlap leaves room for tiny edits (a single cite marker, a timestamp).\n", + "# Set to 1.0 to disable fuzzy dedup entirely (then only exact byte-hashes dedup).\n", + "# Lower to 0.90 to be more aggressive (riskier -- might merge docs that share a lot\n", + "# of boilerplate but ARE different).\n", + "DEDUP_SIMILARITY_THRESHOLD = 0.97\n", + "DEDUP_CHUNK_SIZE = 500 # chars per fragment in the overlapping-fragment signature\n", + "\n", + "# Optional Mammoth style map. Maps Word paragraph styles to HTML headings\n", + "# so mammoth emits real

/

/

instead of generic

. The converter's\n", + "# collapse_extra_h1() then keeps the first H1 (the doc title) and demotes any\n", + "# additional H1s to H2 so every file has exactly ONE H1 (required by Open WebUI's\n", + "# Markdown Header Splitter, which treats each H1 as a new document section).\n", + "CUSTOM_STYLE_MAP = \"\"\"\n", + "p[style-name='Title'] => h1:fresh\n", + "p[style-name='Subtitle'] => h2:fresh\n", + "p[style-name='Heading 1'] => h1:fresh\n", + "p[style-name='Heading 2'] => h2:fresh\n", + "p[style-name='Heading 3'] => h3:fresh\n", + "p[style-name='Heading 4'] => h4:fresh\n", + "\"\"\"\n", + "\n", + "# If True, images are extracted to a per-doc `media/` folder and referenced\n", + "# in Markdown as !media/. Open WebUI currently indexes text only, so\n", + "# inlined image tags are harmless but won't add embedding signal.\n", + "EXTRACT_IMAGES = True" + ] + }, + { + "cell_type": "markdown", + "id": "8f947cd2", + "metadata": {}, + "source": [ + "## 4. Helpers — manifest, hash, URL extraction\n", + "\n", + "These functions handle the persistence bookkeeping:\n", + "\n", + "- `sha256_of_file` streams the file in 1MB chunks (safe for big docx).\n", + "- `load_manifest` / `append_manifest_row` keep `manifest.csv` append-only and Sheets-friendly (`newline=''`).\n", + "- `extract_urls` pulls every full `http(s)://` URL out of converted markdown (no bare-domain guessing — research citations use full URLs).\n", + "- `classify_url` sorts each URL into `pdf` / `html` / `other` by its path extension. The downloader cell does a HEAD request later for Content-Type confirmation; this lightweight classifier is for the per-run summary." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9eda6cb9", + "metadata": {}, + "outputs": [], + "source": [ + "import csv, hashlib, re, datetime\n", + "from pathlib import Path\n", + "\n", + "MANIFEST_COLS = ['sha256', 'first_seen_filename', 'first_seen_at',\n", + " 'last_processed_at', 'input_ext', 'output_path',\n", + " 'output_bytes', 'status']\n", + "SOURCES_COLS = ['doc_sha256', 'doc_stem', 'url', 'url_type']\n", + "DOWNLOADS_COLS = ['url', 'sha256_of_bytes', 'saved_path',\n", + " 'http_status', 'content_type', 'downloaded_at']\n", + "\n", + "def sha256_of_file(path: Path) -> str:\n", + " \"\"\"Stream-hash a file in 1MB chunks; returns hex digest.\"\"\"\n", + " h = hashlib.sha256()\n", + " with open(path, 'rb') as f:\n", + " for chunk in iter(lambda: f.read(1024 * 1024), b''):\n", + " h.update(chunk)\n", + " return h.hexdigest()\n", + "\n", + "def _now_iso() -> str:\n", + " return datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds')\n", + "\n", + "def load_manifest() -> list:\n", + " if not MANIFEST_PATH.exists():\n", + " return []\n", + " with open(MANIFEST_PATH, 'r', newline='') as f:\n", + " return list(csv.DictReader(f))\n", + "\n", + "def append_manifest_row(row: dict):\n", + " exists = MANIFEST_PATH.exists()\n", + " with open(MANIFEST_PATH, 'a', newline='') as f:\n", + " w = csv.DictWriter(f, fieldnames=MANIFEST_COLS)\n", + " if not exists:\n", + " w.writeheader()\n", + " w.writerow({k: row.get(k, '') for k in MANIFEST_COLS})\n", + "\n", + "def is_seen(sha256: str, manifest: list):\n", + " for r in manifest:\n", + " if r.get('sha256') == sha256:\n", + " return r\n", + " return None\n", + "\n", + "_URL_RE = re.compile(r'https?://[^\\s\\)\\]\\>\\}\\,]+')\n", + "def extract_urls(text: str) -> list:\n", + " \"\"\"Extract full http(s):// URLs, stripping trailing punctuation.\"\"\"\n", + " urls, seen = [], set()\n", + " for m in _URL_RE.findall(text):\n", + " u = m.rstrip('.,);:>]}>')\n", + " if u and u not in seen:\n", + " seen.add(u)\n", + " urls.append(u)\n", + " return urls\n", + "\n", + "def classify_url(url: str) -> str:\n", + " \"\"\"Classify a URL by its path extension (no network call).\"\"\"\n", + " path = url.split('?')[0].split('#')[0].lower()\n", + " if path.endswith('.pdf'):\n", + " return 'pdf'\n", + " return 'html'\n", + "\n", + "def append_source_rows(doc_sha256: str, doc_stem: str, urls: list):\n", + " exists = SOURCES_PATH.exists()\n", + " with open(SOURCES_PATH, 'a', newline='') as f:\n", + " w = csv.DictWriter(f, fieldnames=SOURCES_COLS)\n", + " if not exists:\n", + " w.writeheader()\n", + " for u in urls:\n", + " w.writerow({'doc_sha256': doc_sha256, 'doc_stem': doc_stem,\n", + " 'url': u, 'url_type': classify_url(u)})\n", + "\n", + "# --- Pre-flight fuzzy-dedup helpers -------------------------------------------\n", + "# Two files count as \"the same document\" when their overlapping-fragment ratio\n", + "# is >= DEDUP_SIMILARITY_THRESHOLD. We extract raw text fast (python-docx for\n", + "# .docx, plain read for .md), normalize away everything that's not 'real text'\n", + "# (whitespace, punctuation, urls, the gemini masthead, leading '# title'), and\n", + "# split into DEDUP_CHUNK_SIZE-character sliding windows keyed by sha-256. The\n", + "# Jaccard similarity of those two sets ~= fraction of overlapping content.\n", + "DEDUP_DROPS_CSV = DRIVE_ROOT / 'dedup_drops.csv'\n", + "DEDUP_DROPS_COLS = ['kept_filename', 'kept_sha256', 'dropped_filename',\n", + " 'dropped_sha256', 'similarity', 'drop_reason', 'dropped_at']\n", + "\n", + "# Lightweight fallback for extracting text from a .docx WITHOUT running mammoth\n", + "# (which is slow and can crash). Uses python-docx (already installed). Returns\n", + "# an empty string on any failure so the file is still hashed and compared but\n", + "# with effectively no content signal.\n", + "try:\n", + " import docx as _pdocx_dedupe\n", + "except Exception:\n", + " _pdocx_dedupe = None\n", + "\n", + "def _raw_text_for_dedup(path: Path) -> str:\n", + " \"\"\"Get a raw-text signal from a .docx or .md for near-duplicate comparison.\n", + "\n", + " On any failure returns '' - the caller will then fall back to byte-hash dedup.\n", + " \"\"\"\n", + " ext = path.suffix.lower()\n", + " try:\n", + " if ext == '.md':\n", + " return path.read_text(encoding='utf-8', errors='replace')\n", + " if ext == '.docx':\n", + " if _pdocx_dedupe is None:\n", + " return ''\n", + " d = _pdocx_dedupe.Document(str(path))\n", + " parts = []\n", + " for p in d.paragraphs:\n", + " t = p.text\n", + " if t:\n", + " parts.append(t)\n", + " # also pull table text (some real content lives there)\n", + " for tbl in d.tables:\n", + " for row in tbl.rows:\n", + " for cell in row.cells:\n", + " for p in cell.paragraphs:\n", + " if p.text:\n", + " parts.append(p.text)\n", + " return '\\n'.join(parts)\n", + " except Exception:\n", + " return ''\n", + " return ''\n", + "\n", + "_NORM_WS = re.compile(r'\\s+')\n", + "_NORM_PUNCT = re.compile(r'[^a-z0-9 ]')\n", + "_URL_STRIP_RE = re.compile(r'https?://\\S+')\n", + "_MASTHEAD_STRIP_RE = re.compile(r'Created By:.*?DATE:[^\\n]*', re.I | re.S)\n", + "\n", + "def _normalize_for_dedup(text: str) -> str:\n", + " \"\"\"Aggressively normalize for robust near-duplicate comparison.\n", + "\n", + " Removes: URLs (they make every research doc look 100% like another),\n", + " the Gemini masthead block, whitespace, and punctuation. Lowercases.\n", + " \"\"\"\n", + " if not text:\n", + " return ''\n", + " text = _URL_STRIP_RE.sub(' ', text)\n", + " text = _MASTHEAD_STRIP_RE.sub(' ', text)\n", + " text = text.lower()\n", + " text = _NORM_PUNCT.sub(' ', text)\n", + " text = _NORM_WS.sub(' ', text).strip()\n", + " return text\n", + "\n", + "def _chunk_set(text: str, chunk_size: int) -> set:\n", + " \"\"\"Build a set of sha-256 hex digests, one per non-empty non-overlapping\n", + " chunk_size-char window. Used as a fast Jaccard signature.\n", + " \"\"\"\n", + " if not text:\n", + " return set()\n", + " chunks = set()\n", + " i = 0\n", + " n = len(text)\n", + " while i < n:\n", + " piece = text[i:i + chunk_size].strip()\n", + " if piece:\n", + " chunks.add(hashlib.sha256(piece.encode('utf-8')).hexdigest())\n", + " i += chunk_size\n", + " return chunks\n", + "\n", + "def _jaccard(a: set, b: set) -> float:\n", + " \"\"\"Jaccard overlap ratio. Returns 0.0 for two empty sets (treat as not-same).\"\"\"\n", + " if not a or not b:\n", + " return 0.0\n", + " inter = len(a & b)\n", + " union = len(a | b)\n", + " return inter / union if union else 0.0\n", + "\n", + "def append_dedup_drop_row(row: dict):\n", + " exists = DEDUP_DROPS_CSV.exists()\n", + " with open(DEDUP_DROPS_CSV, 'a', newline='') as f:\n", + " w = csv.DictWriter(f, fieldnames=DEDUP_DROPS_COLS)\n", + " if not exists:\n", + " w.writeheader()\n", + " w.writerow({k: row.get(k, '') for k in DEDUP_DROPS_COLS})\n" + ] + }, + { + "cell_type": "markdown", + "id": "209d68fa", + "metadata": {}, + "source": [ + "## 5. Conversion helpers (mammoth pipeline + cleanup)\n", + "\n", + "Preserved verbatim from the tested notebook. These functions chain together to turn a Word doc into clean Markdown with exactly one H1 and good H2/H3 structure. The `convert_one()` function is the docx entry point; `_convert_via_python_docx()` is the fallback when mammoth crashes (e.g. the `_accept0` bug). `clean_md_input()` is new — it runs the same cleanup pipeline on raw uploaded `.md` files (no mammoth pass needed)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f4742867", + "metadata": {}, + "outputs": [], + "source": [ + "import mammoth\n", + "from markdownify import markdownify as md\n", + "import re\n", + "from pathlib import Path\n", + "\n", + "def strip_masthead(text: str) -> str:\n", + " \"\"\"Remove the Gemini Notebook authorship block.\n", + "\n", + " Every Gemini-authored DOCX starts with a masthead like:\n", + " —\n", + " Created By: \n", + " APP: Gemini Notebook\n", + " DATE: \n", + " —\n", + " (sometimes bold-wrapped). It is pure authorship metadata with zero retrieval\n", + " value; we drop it along with its surrounding standalone em-dash dividers.\n", + " Pattern-based so it works regardless of where it sits in the pipeline.\n", + " \"\"\"\n", + " lines = text.split('\\n')\n", + " MASTHEAD_RE = re.compile(r'^(\\*\\*)?(Created By:|APP:\\s*Gemini|DATE:)', re.I)\n", + " EMDASH_RE = re.compile(r'^(\\*\\*)?—(\\*\\*)?$')\n", + " drop_idx = set()\n", + " for i, ln in enumerate(lines):\n", + " s = ln.strip()\n", + " if MASTHEAD_RE.match(s):\n", + " # drop this masthead line, plus adjacent em-dash dividers\n", + " drop_idx.add(i)\n", + " # scan outward for wrapping em-dashes / blank-bounded dashes\n", + " j = i - 1\n", + " while j >= 0 and (lines[j].strip() == '' or EMDASH_RE.match(lines[j].strip())):\n", + " if EMDASH_RE.match(lines[j].strip()):\n", + " drop_idx.add(j)\n", + " break\n", + " j -= 1\n", + " j = i + 1\n", + " while j < len(lines) and (lines[j].strip() == '' or EMDASH_RE.match(lines[j].strip())):\n", + " if EMDASH_RE.match(lines[j].strip()):\n", + " drop_idx.add(j)\n", + " break\n", + " j += 1\n", + " # also fold in multi-line masthead (e.g. 'Created By: X\\nAPP: Y' wrapped in **)\n", + " # by scanning forward through consecutive masthead/em-dash/blank lines\n", + " k = i + 1\n", + " while k < len(lines):\n", + " ks = lines[k].strip()\n", + " if MASTHEAD_RE.match(ks) or EMDASH_RE.match(ks) or ks == '':\n", + " if MASTHEAD_RE.match(ks) or EMDASH_RE.match(ks):\n", + " drop_idx.add(k)\n", + " k += 1\n", + " else:\n", + " break\n", + " if not drop_idx:\n", + " return text\n", + " out = [ln for idx, ln in enumerate(lines) if idx not in drop_idx]\n", + " # collapse any triple+ blanks created by removal\n", + " return re.sub(r'\\n{3,}', '\\n\\n', '\\n'.join(out))\n", + "\n", + "def _make_converter(extract_images: bool, out_dir: Path, doc_stem: str):\n", + " \"\"\"Build a mammoth image converter that saves media to disk.\"\"\"\n", + " if not extract_images:\n", + " return None\n", + " media_dir = out_dir / doc_stem / 'media'\n", + " media_dir.mkdir(parents=True, exist_ok=True)\n", + " counter = {'n': 0}\n", + "\n", + " def convert_image(image):\n", + " counter['n'] += 1\n", + " ext = (image.content_type or 'image/png').split('/')[-1].replace('jpeg', 'jpg')\n", + " fname = f\"image_{counter['n']:03d}.{ext}\"\n", + " with image.open() as f:\n", + " (media_dir / fname).write_bytes(f.read())\n", + " rel = f\"{doc_stem}/media/{fname}\"\n", + " return {'src': rel}\n", + " return convert_image\n", + "\n", + "def clean_markdown(text: str) -> str:\n", + " \"\"\"Remove Word conversion artifacts and normalize whitespace.\"\"\"\n", + " text = re.sub(r']*>', '', text)\n", + " text = re.sub(r']*?)>', '', text)\n", + " text = text.replace('', '')\n", + " text = re.sub(r'\\n{3,}', '\\n\\n', text)\n", + " text = '\\n'.join(line.rstrip() for line in text.splitlines())\n", + " text = re.sub(r'(?m)^#{1,6}\\s*$', '', text)\n", + " return text.strip() + '\\n'\n", + "\n", + "def strip_cite_markers(text: str) -> str:\n", + " \"\"\"Remove Gemini Notebook citation artifacts like '[cite: 1, 2, 3]'.\n", + " These reference internal citation numbers that mean nothing to an LLM\n", + " or embedding model and just waste tokens / add noise to retrieval.\"\"\"\n", + " return re.sub(r'\\s*\\[cite:\\s*[\\d,\\s]+\\]', '', text)\n", + "\n", + "def collapse_hr(text: str) -> str:\n", + " \"\"\"Replace 20+ dash horizontal-rule lines with a sentinel '

' we can\n", + " key section boundaries off of, and drop them as content (a bare HR would\n", + " just become dead weight inside Open WebUI chunks).\"\"\"\n", + " return re.sub(r'(?m)^-{20,}$', '
', text)\n", + "\n", + "def promote_structure(text: str) -> str:\n", + " \"\"\"Promote implicit Word structure into real Markdown headings.\n", + "\n", + " These Gemini-authored DOCX files use bold-only paragraphs (NOT Word\n", + " 'Heading 1/2' styles) for section titles, and a row of dashes as a\n", + " top-level section divider. Mammoth therefore emits them as /bold,\n", + " and Open WebUI's Markdown Header Splitter would have nothing to split on.\n", + "\n", + " Rules:\n", + " - A standalone line fully wrapped in **...** (and NOT inside a table,\n", + " i.e. not starting with '|'), <=160 chars and not a bare '—'\n", + " -> candidate heading.\n", + " - A bold heading immediately after a former HR -> '## ' (H2 section).\n", + " - Any other bold heading -> '### ' (H3 subsection).\n", + " - Table-cell bold stays inside '| ... |' rows and is never promoted.\n", + " - collapse_extra_h1() (later in the pipeline) is what guarantees\n", + " exactly one H1 per file, by keeping the first real H1 (or by\n", + " convert_one prepending a synthetic '# ' if none exists).\n", + " \"\"\"\n", + " lines = text.split('\\n')\n", + " out = []\n", + " after_hr = False\n", + " BOLD_RE = re.compile(r'^\\*\\*(.+?)\\*\\*$')\n", + " for ln in lines:\n", + " stripped = ln.strip()\n", + " if stripped == '
':\n", + " after_hr = True\n", + " continue\n", + " if not stripped:\n", + " out.append(ln)\n", + " continue\n", + " m = BOLD_RE.match(stripped)\n", + " if m and not stripped.startswith('|'):\n", + " inner = m.group(1).strip()\n", + " # collapse adjacent '**A** **B** **C**' style runs into one title\n", + " inner = re.sub(r'\\*\\*\\s*\\*\\*', ' ', inner)\n", + " inner = re.sub(r'\\s+', ' ', inner).strip()\n", + " if len(inner) <= 160 and inner != '—':\n", + " # Promote this bold paragraph to a heading. The first bold line is\n", + " # NOT special-cased anymore: collapse_extra_h1() ensures exactly one\n", + " # H1 per file (the doc's real Title style or the synthetic '# '),\n", + " # so dropping the first bold would just lose a legitimate section\n", + " # header like 'Alpha Section'.\n", + " level = '##' if after_hr else '###'\n", + " out.append(f'{level} {inner}')\n", + " after_hr = False\n", + " continue\n", + " after_hr = False\n", + " out.append(ln)\n", + " return '\\n'.join(out)\n", + "\n", + "def collapse_extra_h1(text: str):\n", + " \"\"\"Ensure exactly one H1 per file.\n", + "\n", + " Some source DOCX files use Word's 'Title' or 'Heading 1' paragraph style, which\n", + " mammoth converts to real

-> '# ' headings. Combined with the synthetic\n", + " '# ' H1 that convert_one prepends, this produces files with 2+ H1s.\n", + " Open WebUI's Markdown Header Splitter treats every H1 as the start of a new\n", + " document section, so multiple H1s fragment retrieval badly.\n", + "\n", + " Returns (new_text, has_h1): if the doc already has at least one real H1, we keep\n", + " the first one and demote any subsequent H1s to H2; the caller then SKIPS prepending\n", + " the synthetic '# ' title (since the doc already has a title H1).\n", + " \"\"\"\n", + " lines = text.split('\\n')\n", + " h1_seen = False\n", + " out = []\n", + " for ln in lines:\n", + " m = re.match(r'^(#{1})\\s+(.*)', ln)\n", + " if m:\n", + " if not h1_seen:\n", + " # Keep the first H1 as the document title.\n", + " h1_seen = True\n", + " out.append(ln)\n", + " else:\n", + " # Demote every subsequent H1 to H2 so there is exactly one H1.\n", + " out.append(f'## {m.group(2)}')\n", + " else:\n", + " out.append(ln)\n", + " return '\\n'.join(out), h1_seen\n", + "\n", + "def convert_one(docx_path: Path, out_root: Path, style_map: str, extract_images: bool):\n", + " \"\"\"Convert a single .docx to .md. Returns (out_file, warnings).\"\"\"\n", + " stem = docx_path.stem\n", + " with open(docx_path, 'rb') as f:\n", + " kwargs = dict()\n", + " if style_map:\n", + " kwargs['style_map'] = style_map\n", + " if extract_images:\n", + " kwargs['convert_image'] = _make_converter(True, out_root, stem)\n", + " result = mammoth.convert_to_html(f, **kwargs)\n", + "\n", + " html = result.value\n", + " markdown = md(html, heading_style='ATX', bullets='-', strip=['a'])\n", + " # Pipeline order matters: strip noise -> collapse HR -> promote structure\n", + " markdown = clean_markdown(markdown)\n", + " markdown = strip_masthead(markdown)\n", + " markdown = strip_cite_markers(markdown)\n", + " markdown = collapse_hr(markdown)\n", + " markdown = promote_structure(markdown)\n", + " markdown, has_h1 = collapse_extra_h1(markdown)\n", + "\n", + " # Synthetic H1 from the source filename stem gives every knowledge-base\n", + " # chunk a stable, cite-friendly document title (Open WebUI shows this).\n", + " if not has_h1:\n", + " # No real H1 in the source -> use the filename stem as the title.\n", + " markdown = f\"# {stem}\\n\\n\" + markdown.strip() + \"\\n\"\n", + " else:\n", + " # Source already has an H1 (e.g. Word Title style); keep it as the doc title\n", + " # and do NOT prepend a duplicate synthetic H1 (would re-introduce the 2x-H1 bug).\n", + " markdown = markdown.strip() + \"\\n\"\n", + "\n", + " out_file = out_root / f\"{stem}.md\"\n", + " out_file.write_text(markdown, encoding='utf-8')\n", + " return out_file, result.messages\n", + "\n", + "# --- Helper: python-docx fallback for corrupt docx that crashes mammoth ----------\n", + "try:\n", + " import docx as _pdocx # python-docx\n", + "except Exception:\n", + " _pdocx = None\n", + "\n", + "def _convert_via_python_docx(docx_path: Path, out_file: Path, stem: str):\n", + " \"\"\"Fallback converter for docx files mammoth crashed on (e.g. the _accept0 bug).\\n \\n Extracts paragraph text + structure via python-docx. Differs from mammoth path:\\n - Recognizes Word Heading 1/2/3/4 + Title styles as real markdown headings.\\n - Detects code/CLI paragraphs (shell prompts, Python comments, etc.) and fences\\n them in ``` blocks so leading \"# comment\" lines do NOT become faux H1 markdown\\n headings (a bug that previously produced files with H1:6 / H1:15 / H1:279).\\n - Runs collapse_extra_h1() as a safety net so the file ends up with exactly\\n one H1, matching the contract of the primary convert_one() path.\\n \"\"\"\n", + " if _pdocx is None:\n", + " raise RuntimeError('python-docx not installed; cannot fallback')\n", + " d = _pdocx.Document(str(docx_path))\n", + " lines = []\n", + " h1_seen = False\n", + " in_code = False # whether we are currently inside a fenced code block\n", + " # Heuristics for detecting a paragraph that is really a code/CLI line, not prose.\n", + " # \"Looks like code\" -> preserve verbatim inside a fenced block so neither the\n", + " # leading '# ' (Python/shell comment) nor leading shell symbols become faux\n", + " # markdown headings.\n", + " import re as _re\n", + " def _looks_like_code(s: str) -> bool:\n", + " if not s:\n", + " return False\n", + " # Common shell prompt prefixes\n", + " if _re.match(r'^(\\$|>#|>|\\$\\s|sudo |apt |pip |git |cd |npm |python|wget |curl |chmod |mkdir |cp |mv |rm |echo |export |systemctl |dpkg |sed |awk |grep |find |tar |unzip |./|npm |npx|docker |kubectl |helm |kubectl )', s):\n", + " return True\n", + " # Python-style comment (# ...) or shell comment. Excludes legitimate prose\n", + " # like '# Topic Idea:' only if it ALSO has code-y characters. To be safe,\n", + " # treat a line starting with '# ' followed by uppercase letters OR containing\n", + " # common code tokens as a code comment. We DON'T classify every '#'-leading\n", + " # line as code (would defeat real H1 detection); we only fence when it is\n", + " # clearly not a title. Real titles are short (<80 chars), Capitalized prose.\n", + " m = _re.match(r'^# (.{1,160})$', s)\n", + " if m and _re.search(r'[;|=&><{}\\\\/]|from \\w|import \\w|def \\w|class \\w|^if |^else|^for |^while |^return |=>|->|name ==', s):\n", + " return True\n", + " return False\n", + " for p in d.paragraphs:\n", + " txt = p.text.rstrip()\n", + " if not txt.strip():\n", + " if in_code:\n", + " # blank line inside a code block: keep it (preserves formatting)\n", + " lines.append('')\n", + " else:\n", + " lines.append('')\n", + " continue\n", + " style = (p.style.name or '').lower() if p.style else ''\n", + " # 1. Word heading styles -> real markdown headings\n", + " if 'heading 1' in style or style == 'title':\n", + " if in_code: lines.append('```'); in_code = False\n", + " if not h1_seen: lines.append(f'# {txt.strip()}'); h1_seen = True\n", + " else: lines.append(f'## {txt.strip()}')\n", + " elif 'heading 2' in style:\n", + " if in_code: lines.append('```'); in_code = False\n", + " lines.append(f'## {txt.strip()}')\n", + " elif 'heading 3' in style:\n", + " if in_code: lines.append('```'); in_code = False\n", + " lines.append(f'### {txt.strip()}')\n", + " elif 'heading 4' in style:\n", + " if in_code: lines.append('```'); in_code = False\n", + " lines.append(f'#### {txt.strip()}')\n", + " else:\n", + " # NOT a styled heading. Two sub-cases:\n", + " # (a) Bold-only short paragraph in a Normal style -> candidate H3 section header\n", + " is_bold = bool(p.runs) and all(r.bold for r in p.runs if r.text.strip())\n", + " if is_bold and len(txt.strip()) <= 160 and not txt.lstrip().startswith('#'):\n", + " if in_code: lines.append('```'); in_code = False\n", + " lines.append(f'### {txt.strip()}')\n", + " elif _looks_like_code(txt.strip()):\n", + " # (b) code/comment line -> accumulate inside a fenced block.\n", + " # Crucial: a '# foo' line inside ``` ``` is NOT a markdown H1.\n", + " if not in_code:\n", + " lines.append('```')\n", + " in_code = True\n", + " lines.append(txt)\n", + " else:\n", + " # plain prose paragraph -> close any open code block first\n", + " if in_code: lines.append('```'); in_code = False\n", + " lines.append(txt.strip())\n", + " if in_code: lines.append('```') # close trailing code block\n", + " body = '\\n'.join(lines).strip()\n", + " if not h1_seen:\n", + " body = f'# {stem}\\n\\n' + body\n", + " # Safety net: ensure exactly one H1 (e.g. if doc has multiple 'Title' paragraphs,\n", + " # or a '# comment' paragraph slipped past the heuristics above).\n", + " body, _ = collapse_extra_h1(body)\n", + " out_file.write_text(body + '\\n', encoding='utf-8')\n", + " return out_file, []\n", + "\n", + "\n", + "\n", + "\n", + "def clean_md_input(path: Path, out_file: Path, stem: str):\n", + " \"\"\"Cleanup pipeline for raw uploaded .md files.\n", + "\n", + " Same post-processing as docx, minus mammoth (the file is already markdown):\n", + " clean_markdown -> strip_masthead -> strip_cite_markers\n", + " -> collapse_hr -> promote_structure -> collapse_extra_h1\n", + " -> conditional synthetic '# {stem}' H1 prepend.\n", + " \"\"\"\n", + " text = path.read_text(encoding='utf-8')\n", + " text = clean_markdown(text)\n", + " text = strip_masthead(text)\n", + " text = strip_cite_markers(text)\n", + " text = collapse_hr(text)\n", + " text = promote_structure(text)\n", + " text, has_h1 = collapse_extra_h1(text)\n", + " if not has_h1:\n", + " text = f\"# {stem}\\n\\n\" + text.strip() + \"\\n\"\n", + " else:\n", + " text = text.strip() + \"\\n\"\n", + " out_file.write_text(text, encoding='utf-8')\n", + " return out_file\n" + ] + }, + { + "cell_type": "markdown", + "id": "ea876def", + "metadata": {}, + "source": [ + "## 5b. Pre-flight: drop near-duplicates by content similarity\n", + "\n", + "Before conversion, look at the inputs in THIS upload batch and drop any file\n", + "whose normalized text overlaps an already-kept file by more than\n", + "`DEDUP_SIMILARITY_THRESHOLD` (default 0.97 — catches `(1)`/`(2)`/`Copy of`\n", + "variants where the bytes differ by a few characters but the document is\n", + "really the same).\n", + "\n", + "**How it works:** extracts raw text (`python-docx` for `.docx`, plain read\n", + "for `.md`), strips URLs + masthead + whitespace + punctuation, slices into\n", + "non-overlapping 500-char windows, hashes each window, and computes the Jaccard\n", + "overlap between every pair. Matches above threshold → drop the second one and\n", + "log it to `dedup_drops.csv`.\n", + "\n", + "The original upload list is snapshotted in cell 2 as `uploaded_files_master`,\n", + "so this cell is **idempotent**: rerun it as many times as you want, it always\n", + "rebuilds `input_files` from the snapshot. Threshold tunable in the Config cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eefa73ce", + "metadata": {}, + "outputs": [], + "source": [ + "# Start from the snapshot so reruns are idempotent.\n", + "input_files = list(uploaded_files_master) if 'uploaded_files_master' in globals() \\\n", + " else list(input_files)\n", + "\n", + "if not DEDUP_SIMILARITY_THRESHOLD or DEDUP_SIMILARITY_THRESHOLD <= 0:\n", + " print(f\"Near-duplicate scan: DISABLED (DEDUP_SIMILARITY_THRESHOLD={DEDUP_SIMILARITY_THRESHOLD})\")\n", + " print(f\"Proceeding with {len(input_files)} input file(s).\")\n", + "else:\n", + " print(f\"Near-duplicate scan (threshold: {DEDUP_SIMILARITY_THRESHOLD:.2f} chunk overlap)...\")\n", + " print(f\"Hashing {len(input_files)} input(s) into normalized text signatures...\")\n", + "\n", + " # Build a normalized signature for each input: raw byte-hash AND a normalized\n", + " # chunk set (used for the fuzzy comparison).\n", + " signatures = []\n", + " for path in input_files:\n", + " sha = sha256_of_file(path)\n", + " raw = _raw_text_for_dedup(path)\n", + " norm = _normalize_for_dedup(raw)\n", + " chunks = _chunk_set(norm, DEDUP_CHUNK_SIZE)\n", + " signatures.append({'path': path, 'sha': sha, 'chunks': chunks,\n", + " 'n_chars': len(norm), 'n_chunks': len(chunks)})\n", + "\n", + " # First pass: group by exact byte hash (instant, free, catches identical uploads).\n", + " by_sha = {}\n", + " for s in signatures:\n", + " by_sha.setdefault(s['sha'], []).append(s)\n", + "\n", + " kept = [] # list of signature dicts we'll keep\n", + " dropped_exact = [] # (kept_sig, dup_sig)\n", + " for sha, group in by_sha.items():\n", + " if len(group) == 1:\n", + " kept.append(group[0])\n", + " else:\n", + " # Keep the first, drop the rest with similarity 1.00 (exact).\n", + " keep = group[0]\n", + " kept.append(keep)\n", + " for dup in group[1:]:\n", + " dropped_exact.append((keep, dup))\n", + "\n", + " # Second pass: fuzzy compare across all 'kept' so far. For each candidate,\n", + " # compare its chunk set against every already-accepted doc whose chunk count\n", + " # is within a sane ratio (10x either way; wildly different doc sizes can't be\n", + " # 97% overlapping unless one is mostly empty, which we skip).\n", + " final_kept = []\n", + " dropped_fuzzy = [] # (kept_sig, dup_sig, similarity)\n", + " for cand in kept:\n", + " if not cand['chunks']:\n", + " # Couldn't extract text (failed docx parse, empty file). Keep it\n", + " # unconditionally; byte-hash dedup already handled exact dupes.\n", + " final_kept.append(cand)\n", + " continue\n", + " match = None\n", + " for keep in final_kept:\n", + " if not keep['chunks']:\n", + " continue\n", + " nc, nk = cand['n_chunks'], keep['n_chunks']\n", + " if nc == 0 or nk == 0:\n", + " continue\n", + " # Cheap pre-filter: can't be 97% similar if chunk counts are >10x apart.\n", + " if nc > 10 * nk or nk > 10 * nc:\n", + " continue\n", + " sim = _jaccard(cand['chunks'], keep['chunks'])\n", + " if sim >= DEDUP_SIMILARITY_THRESHOLD:\n", + " match = (keep, sim)\n", + " break\n", + " if match:\n", + " dropped_fuzzy.append((match[0], cand, match[1]))\n", + " else:\n", + " final_kept.append(cand)\n", + "\n", + " # Build the new input_files list, preserving the original iteration order.\n", + " kept_paths = {s['path'] for s in final_kept}\n", + " input_files = [p for p in input_files if p in kept_paths]\n", + "\n", + " # Report\n", + " print()\n", + " print(f\"Exact-hash duplicates dropped: {len(dropped_exact)}\")\n", + " for keep, dup in dropped_exact:\n", + " print(f\" Keep '{keep['path'].name}'\")\n", + " print(f\" ⤷ dup '{dup['path'].name}' — identical bytes — dropped\")\n", + " append_dedup_drop_row({\n", + " 'kept_filename': keep['path'].name, 'kept_sha256': keep['sha'],\n", + " 'dropped_filename': dup['path'].name, 'dropped_sha256': dup['sha'],\n", + " 'similarity': '1.000', 'drop_reason': 'exact_byte_match',\n", + " 'dropped_at': _now_iso()})\n", + "\n", + " print()\n", + " print(f\"Near-duplicate (fuzzy) drops: {len(dropped_fuzzy)}\")\n", + " for keep, dup, sim in dropped_fuzzy:\n", + " print(f\" Keep '{keep['path'].name}' ({keep['n_chunks']} chunks)\")\n", + " print(f\" ⤷ dup '{dup['path'].name}' — {sim*100:.1f}% overlap — dropped\")\n", + " append_dedup_drop_row({\n", + " 'kept_filename': keep['path'].name, 'kept_sha256': keep['sha'],\n", + " 'dropped_filename': dup['path'].name, 'dropped_sha256': dup['sha'],\n", + " 'similarity': f\"{sim:.3f}\", 'drop_reason': 'fuzzy_overlap',\n", + " 'dropped_at': _now_iso()})\n", + "\n", + " print()\n", + " print(f\"Deduped: {len(uploaded_files_master)} → {len(input_files)} inputs \"\n", + " f\"({len(uploaded_files_master) - len(input_files)} duplicates dropped).\")\n", + " print(f\"Pre-flight complete. Cell 6 will now convert {len(input_files)} file(s).\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "5b60771c", + "metadata": {}, + "source": [ + "## 6. Convert all inputs (with manifest dedup)\n", + "\n", + "For each input file:\n", + "1. Compute SHA-256 of the **original input bytes** (before any conversion).\n", + "2. Check the manifest. If the hash is already there and `SKIP_ALREADY_SEEN` is True → print `SKIP` and move on. If you renamed a file, the hash still matches → no duplicate convert. If the bytes *actually* differ → different hash → converts as a new doc.\n", + "3. Dispatch by extension: `.docx` → `convert_one()` with python-docx fallback; `.md` → `clean_md_input()`.\n", + "4. Copy the original bytes to `archive/` (so future Colab sessions can still match the hash even after `/content` is wiped).\n", + "5. Append a row to `manifest.csv` with the status: `converted` / `crashed_fallback_ok` / `crashed_skipped`.\n", + "\n", + "The `converted_this_run` list is what the verify and URL-extraction cells iterate over — they only look at files WE converted in this run, not the entire `markdown_output/` folder (so previously-converted files don't drown out the per-run summary)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5095483b", + "metadata": {}, + "outputs": [], + "source": [ + "manifest = load_manifest()\n", + "manifest_seen = {r['sha256'] for r in manifest}\n", + "converted_this_run = [] # list of (out_path, sha256, stem)\n", + "skipped_count = 0\n", + "crashed_count = 0\n", + "\n", + "for path in input_files:\n", + " sha = sha256_of_file(path)\n", + " ext = path.suffix.lower().lstrip('.')\n", + " stem = path.stem\n", + "\n", + " # Dedup check against the persistent Drive manifest. Guard against the rare\n", + " # race where sha is in manifest_seen (the set) but is_seen() returns None\n", + " # (e.g. another process appended to manifest.csv after we cached the set).\n", + " if SKIP_ALREADY_SEEN and sha in manifest_seen:\n", + " seen_row = is_seen(sha, manifest) or {}\n", + " prev_name = seen_row.get('first_seen_filename', '(unknown)')\n", + " prev_date = seen_row.get('first_seen_at', '(unknown)')\n", + " print(f\"SKIP {path.name} (already converted; first seen as \"\n", + " f\"'{prev_name}' on {prev_date})\")\n", + " skipped_count += 1\n", + " continue\n", + "\n", + " # Archive original input bytes (durable across Colab wipes)\n", + " if ARCHIVE_INPUTS:\n", + " archive_path = ARCHIVE_DIR / f\"{sha}.{ext}\"\n", + " if not archive_path.exists():\n", + " shutil.copy2(path, archive_path)\n", + "\n", + " out_file = MD_OUTPUT_DIR / f\"{stem}.md\"\n", + " status = 'converted'\n", + " try:\n", + " if ext == 'docx':\n", + " try:\n", + " out_file, warnings = convert_one(path, MD_OUTPUT_DIR,\n", + " CUSTOM_STYLE_MAP, EXTRACT_IMAGES)\n", + " except Exception:\n", + " out_file, warnings = _convert_via_python_docx(path, out_file, stem)\n", + " status = 'crashed_fallback_ok'\n", + " elif ext == 'md':\n", + " out_file = clean_md_input(path, out_file, stem)\n", + " status = 'converted'\n", + " else:\n", + " print(f\"SKIP {path.name} (unsupported extension .{ext})\")\n", + " continue\n", + "\n", + " size = out_file.stat().st_size\n", + " rel_out = str(out_file.relative_to(DRIVE_ROOT))\n", + " row = {\n", + " 'sha256': sha,\n", + " 'first_seen_filename': path.name,\n", + " 'first_seen_at': _now_iso(),\n", + " 'last_processed_at': _now_iso(),\n", + " 'input_ext': ext,\n", + " 'output_path': rel_out,\n", + " 'output_bytes': size,\n", + " 'status': status,\n", + " }\n", + " append_manifest_row(row)\n", + " manifest_seen.add(sha)\n", + " converted_this_run.append((out_file, sha, stem))\n", + " print(f\"✓ {path.name} → {out_file.name} ({size:,} bytes) [{status}]\")\n", + " except Exception as e:\n", + " crashed_count += 1\n", + " row = {\n", + " 'sha256': sha,\n", + " 'first_seen_filename': path.name,\n", + " 'first_seen_at': _now_iso(),\n", + " 'last_processed_at': _now_iso(),\n", + " 'input_ext': ext,\n", + " 'output_path': '',\n", + " 'output_bytes': 0,\n", + " 'status': 'crashed_skipped',\n", + " }\n", + " append_manifest_row(row)\n", + " print(f\"✗ {path.name} FAILED ({type(e).__name__}: {e})\")\n", + "\n", + "print(f\"\\nDone. {len(converted_this_run)} converted, \"\n", + " f\"{skipped_count} skipped (already in manifest), \"\n", + " f\"{crashed_count} crashed.\")\n", + "print(f\"Manifest now has {len(load_manifest())} total rows.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "0ada1c03", + "metadata": {}, + "source": [ + "## 7. Extract source URLs from converted files\n", + "\n", + "For every file converted THIS run, pull every full `http(s)://` URL out of the markdown and append a row to `sources.csv`. Each row records the source doc’s SHA + stem alongside the URL and a lightweight type guess (`pdf` / `html`). This is the list the optional downloader cell consumes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8d9beedd", + "metadata": {}, + "outputs": [], + "source": [ + "total_urls = 0\n", + "docs_with_urls = 0\n", + "for out_path, sha, stem in converted_this_run:\n", + " text = out_path.read_text(encoding='utf-8')\n", + " urls = extract_urls(text)\n", + " if urls:\n", + " append_source_rows(sha, stem, urls)\n", + " total_urls += len(urls)\n", + " docs_with_urls += 1\n", + " types = [classify_url(u) for u in urls]\n", + " n_pdf = types.count('pdf')\n", + " n_html = types.count('html')\n", + " n_other = types.count('other')\n", + " print(f\" {stem[:70]}: {len(urls)} URLs ({n_pdf} pdf, {n_html} html, {n_other} other)\")\n", + " else:\n", + " print(f\" {stem[:70]}: 0 URLs\")\n", + "\n", + "print(f\"\\nTotal: {total_urls} URLs across {docs_with_urls} docs \"\n", + " f\"({len(converted_this_run) - docs_with_urls} had no URLs).\")" + ] + }, + { + "cell_type": "markdown", + "id": "6973783c", + "metadata": {}, + "source": [ + "## 8. Verify heading structure (important for Open WebUI chunking)\n", + "\n", + "Open WebUI's **Markdown Header Splitter** chunks on `#`/`##`/`###`. Run this cell to confirm each converted file has **exactly one H1** plus a healthy H2/H3 hierarchy. The audit only runs over files WE converted this run (so reruns aren't drowned by previously-converted files in `markdown_output/`).\n", + "\n", + "Flags:\n", + "- `⚠ multiple H1s!` — would fragment retrieval; collapse_extra_h1 should prevent this.\n", + "- `⚠ nearly empty` — output is suspiciously small; conversion may have failed silently.\n", + "- `⚠ NO sub-headings!` — the document would collapse into a single giant chunk." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2c790d68", + "metadata": {}, + "outputs": [], + "source": [ + "from collections import Counter\n", + "import re\n", + "\n", + "print(\"Heading structure per converted file (this run only):\\n\")\n", + "for out_path, sha, stem in converted_this_run:\n", + " counts = Counter()\n", + " text = out_path.read_text(encoding='utf-8')\n", + " for ln in text.splitlines():\n", + " m = re.match(r'^(#{1,6})\\s', ln)\n", + " if m:\n", + " counts[len(m.group(1))] += 1\n", + " summary = \", \".join(f\"H{lvl}:{counts[lvl]}\" for lvl in sorted(counts))\n", + " flag_no = \" ⚠ NO sub-headings!\" if counts.get(2, 0) + counts.get(3, 0) == 0 else \"\"\n", + " flag_size = \" ⚠ nearly empty\" if len(text) < 500 else \"\"\n", + " flag_h1 = \" ⚠ multiple H1s!\" if counts.get(1, 0) > 1 else \"\"\n", + " print(f\" {stem[:70]}\")\n", + " print(f\" {summary} ({len(text):,} chars){flag_no}{flag_size}{flag_h1}\")\n", + "\n", + "print(f\"\\nManifest total: {len(load_manifest())} files tracked since first run.\")" + ] + }, + { + "cell_type": "markdown", + "id": "7d4750ec", + "metadata": {}, + "source": [ + "## 9. Preview a converted file\n", + "\n", + "Quick sanity check. Change the index to preview a different file. Glob is filtered by `is_file()` so we don't trip on `media/` subdirectories." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9647fd57", + "metadata": {}, + "outputs": [], + "source": [ + "preview_idx = 0 # change to view another file\n", + "md_files = sorted(p for p in MD_OUTPUT_DIR.glob('*.md') if p.is_file())\n", + "assert md_files, \"No .md files produced.\"\n", + "target = md_files[preview_idx]\n", + "print(f\"# Preview of: {target.name}\\n\" + \"=\" * 60)\n", + "content = target.read_text(encoding='utf-8')\n", + "print(content[:3000])\n", + "if len(content) > 3000:\n", + " print(f\"\\n… ({len(content)-3000:,} more chars)\")" + ] + }, + { + "cell_type": "markdown", + "id": "515a7730", + "metadata": {}, + "source": [ + "## 10. Bundle as `.zip` and download\n", + "\n", + "Zips the converted `.md` files (and their `media/` folders) into a local archive and triggers a browser download. Useful on free tier where the VM disappears after the session — but since everything is also saved to Drive (`markdown_output/`), you don't strictly need this." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6422bd2c", + "metadata": {}, + "outputs": [], + "source": [ + "import shutil\n", + "from google.colab import files as colab_files\n", + "\n", + "ZIP_PATH = Path('/content/markdown_for_openwebui.zip')\n", + "if ZIP_PATH.exists():\n", + " ZIP_PATH.unlink()\n", + "\n", + "shutil.make_archive('/content/markdown_for_openwebui', 'zip',\n", + " root_dir=str(MD_OUTPUT_DIR))\n", + "print(f\"Created {ZIP_PATH} ({ZIP_PATH.stat().st_size:,} bytes)\")\n", + "\n", + "colab_files.download(str(ZIP_PATH))" + ] + }, + { + "cell_type": "markdown", + "id": "27801831", + "metadata": {}, + "source": [ + "## 11. (Optional) Bulk URL downloader — OFF by default\n", + "\n", + "Reads `sources.csv`, skips any URL already in `downloads.csv`, makes a HEAD request to determine the real Content-Type, then saves the response bytes to:\n", + "\n", + "- `downloads/pdf/.pdf` if Content-Type is PDF or URL ends in `.pdf`\n", + "- `downloads/html/.html` if Content-Type is HTML\n", + "- `downloads/other/.bin` otherwise\n", + "\n", + "**Paywalled / auth-locked URLs (HTTP 401/403/etc.) are recorded with an empty `saved_path`** so you can see at a glance what's unreachable. The downloader will not try to bypass paywalls.\n", + "\n", + "To actually run the downloader, **flip `RUN_DOWNLOADER = True`** in the cell below. It defaults to `False` so a \"Run all\" doesn't kick off hundreds of HTTP requests." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5edc7ac0", + "metadata": {}, + "outputs": [], + "source": [ + "RUN_DOWNLOADER = False # <-- flip to True to actually run the downloader\n", + "\n", + "if RUN_DOWNLOADER:\n", + " import requests, csv, hashlib, time\n", + " from pathlib import Path\n", + "\n", + " # Make sure downloads.csv has a header if it's new.\n", + " if not DOWNLOADS_CSV.exists():\n", + " with open(DOWNLOADS_CSV, 'w', newline='') as f:\n", + " csv.DictWriter(f, fieldnames=DOWNLOADS_COLS).writeheader()\n", + "\n", + " # Load already-downloaded URLs\n", + " already = set()\n", + " with open(DOWNLOADS_CSV, 'r', newline='') as f:\n", + " for r in csv.DictReader(f):\n", + " already.add(r['url'])\n", + "\n", + " if not SOURCES_PATH.exists():\n", + " print(\"No sources.csv found. Run the URL extraction cell first.\")\n", + " else:\n", + " with open(SOURCES_PATH, 'r', newline='') as f:\n", + " source_rows = list(csv.DictReader(f))\n", + " unique_urls = sorted({r['url'] for r in source_rows})\n", + " todo = [u for u in unique_urls if u not in already]\n", + " print(f\"{len(unique_urls)} unique URLs in sources.csv; \"\n", + " f\"{len(already)} already downloaded; \"\n", + " f\"will fetch {len(todo)}.\")\n", + "\n", + " fetched = 0\n", + " failed = 0\n", + " for url in unique_urls:\n", + " if url in already:\n", + " continue\n", + " try:\n", + " # HEAD first to get content-type and final URL.\n", + " h = requests.head(url, allow_redirects=True, timeout=15,\n", + " headers={'User-Agent': 'Mozilla/5.0'})\n", + " ctype = h.headers.get('Content-Type', '').lower()\n", + " if 'pdf' in ctype or url.lower().endswith('.pdf'):\n", + " kind, ext = 'pdf', 'pdf'\n", + " elif 'html' in ctype:\n", + " kind, ext = 'html', 'html'\n", + " else:\n", + " kind, ext = 'other', 'bin'\n", + "\n", + " if not (200 <= h.status_code < 300):\n", + " with open(DOWNLOADS_CSV, 'a', newline='') as f:\n", + " csv.DictWriter(f, fieldnames=DOWNLOADS_COLS).writerow({\n", + " 'url': url, 'sha256_of_bytes': '',\n", + " 'saved_path': '', 'http_status': h.status_code,\n", + " 'content_type': ctype, 'downloaded_at': _now_iso()})\n", + " print(f\"✗ {url} → HTTP {h.status_code} (skipped)\")\n", + " failed += 1\n", + " time.sleep(RATE_LIMIT_SEC)\n", + " continue\n", + "\n", + " # Fetch the actual bytes.\n", + " r = requests.get(url, allow_redirects=True, timeout=30,\n", + " headers={'User-Agent': 'Mozilla/5.0'})\n", + " if not (200 <= r.status_code < 300):\n", + " with open(DOWNLOADS_CSV, 'a', newline='') as f:\n", + " csv.DictWriter(f, fieldnames=DOWNLOADS_COLS).writerow({\n", + " 'url': url, 'sha256_of_bytes': '',\n", + " 'saved_path': '', 'http_status': r.status_code,\n", + " 'content_type': r.headers.get('Content-Type', ''),\n", + " 'downloaded_at': _now_iso()})\n", + " print(f\"✗ {url} → HTTP {r.status_code}\")\n", + " failed += 1\n", + " time.sleep(RATE_LIMIT_SEC)\n", + " continue\n", + "\n", + " body_sha = hashlib.sha256(r.content).hexdigest()\n", + " saved = DOWNLOADS_DIR / kind / f\"{body_sha}.{ext}\"\n", + " saved.write_bytes(r.content)\n", + " rel = str(saved.relative_to(DRIVE_ROOT))\n", + " with open(DOWNLOADS_CSV, 'a', newline='') as f:\n", + " csv.DictWriter(f, fieldnames=DOWNLOADS_COLS).writerow({\n", + " 'url': url, 'sha256_of_bytes': body_sha,\n", + " 'saved_path': rel, 'http_status': r.status_code,\n", + " 'content_type': r.headers.get('Content-Type', ''),\n", + " 'downloaded_at': _now_iso()})\n", + " fetched += 1\n", + " print(f\"✓ {url} → {rel} ({len(r.content):,} bytes)\")\n", + " time.sleep(RATE_LIMIT_SEC)\n", + " except Exception as e:\n", + " print(f\"✗ {url} → {type(e).__name__}: {e}\")\n", + " failed += 1\n", + " time.sleep(RATE_LIMIT_SEC)\n", + "\n", + " print(f\"\\nDone. Fetched {fetched} new, failed/skipped {failed}.\")\n", + "else:\n", + " print(\"Downloader is OFF. Set RUN_DOWNLOADER = True above to run.\")" + ] + }, + { + "cell_type": "markdown", + "id": "e454beab", + "metadata": {}, + "source": [ + "## 12. Import into Open WebUI (manual one-time step)\n", + "\n", + "After unzipping locally:\n", + "\n", + "1. Open your Open WebUI instance → **Workspace → Knowledge**.\n", + "2. Create or open a knowledge base.\n", + "3. Upload the `.md` files (image folders are not currently indexed by Open WebUI's text RAG — they're safe to omit unless you want them stored alongside).\n", + "4. In **Admin Settings → Tools → Documents**, make sure **Markdown Header Splitting** is **enabled**.\n", + "5. Set sensible chunking values, per the Open WebUI docs:\n", + " - **Chunk Size**: ~1000–2000 characters.\n", + " - **Chunk Overlap**: ~10–15% of chunk size.\n", + " - **Chunk Min Size Target**: ~50% of chunk size (merges tiny header fragments).\n", + "6. If you change embedding models later, hit **Reindex** so all chunks are re-embedded with the new model. Old embeddings live in a different vector space and won't retrieve well until re-indexed.\n", + "\n", + "> These config recommendations come from the Open WebUI RAG documentation [2]; the notebook itself only handles the DOCX→Markdown conversion." + ] + } + ], + "metadata": { + "accelerator": "None", + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/07-tasks/docx-to-markdown-converter-notebook.md b/07-tasks/docx-to-markdown-converter-notebook.md new file mode 100644 index 0000000..f7d13c8 --- /dev/null +++ b/07-tasks/docx-to-markdown-converter-notebook.md @@ -0,0 +1,102 @@ +--- +title: "Task: DOCX→Markdown converter notebook for Open WebUI" +status: "active" +folder: "07-tasks" +tags: [task, notebook, openwebui, docx, markdown, converter] +created: "2026-07-30" +updated: "2026-07-30" +version: "1.0.0" +priority: "high" +due: "2026-07-30" +owner: "stateofshit" +--- + +# Task: DOCX→Markdown converter notebook for Open WebUI + +## Done — 2026-07-30 + +## What + +A Colab notebook that batches `.docx` (and `.md`) inputs into clean Markdown optimized for embedding in **Open WebUI** knowledge bases. Open WebUI's RAG uses Markdown Header Splitting, so the converter preserves a clean single-H1 / nested-H2-H3 heading hierarchy — that's what makes retrieval chunks good. Output is mirrored to Google Drive so nothing is lost across Colab wipes. + +This notebook is the upgrade over the older `/home/user/00-incoming/docx_to_markdown.ipynb`: it adds the Gemini-Notebook setup nuance (cite markers stripped, hard bold-promoted to headings) and the dedup edge-cases we hit this week. + +## Pipeline (per file) + +- **DOCX**: `mammoth(html) → clean_markdown → strip_masthead → strip_cite_markers → collapse_hr → promote_structure → collapse_extra_h1 → conditional synthetic H1 prepend` +- **MD** (raw uploaded): same chain minus mammoth +- **Fallback**: if mammoth crashes (the `_accept0` bug on weird docx), the pipeline degrades to a `python-docx` parser that still produces clean markdown +- **Output**: one `.md` per input → `/drive/MyDrive/to_convert/markdown_output/` + +## Headlining features + +| # | Feature | Why it matters | +|---|---|---| +| 1 | Drive-backed manifest dedup (SHA-256 of input bytes) | Renaming a file does NOT cause a reconvert. Survives Colab wipes. | +| 2 | **Pre-flight fuzzy dedup** (new today — cell 5b) | Drops `(1)`/`(2)`/`Copy of` variants where the bytes differ by a few chars but the doc is really the same. Uses Jaccard similarity over 500-char normalized text chunks. Threshold `0.97` (tunable). | +| 3 | Gemini masthead + cite-marker stripping | Removes `— / Created By: Due Process Demolition / APP: Gemini Notebook / DATE: …` boilerplate and `[cite: N]` artifacts so the KB doesn't fill with garbage | +| 4 | Auto heading promotion | For Gemini-Notebook-style docs that use bold paragraphs (not Word heading styles) for section titles, the converter promotes `**Bold**` lines to real `##`/`###` so Open WebUI's splitter has structure | +| 5 | Single-H1 enforcement | `collapse_extra_h1()` keeps the first H1 (title) and demotes any subsequent H1 → H2. Required by Open WebUI's Markdown Header Splitter | +| 6 | URL extraction | Every `http(s)://` URL found in converted .md files → `sources.csv` paired with source-doc SHA + stem | +| 7 | Bulk downloader (off by default, cell 11) | Once sources.csv is curated, opt-in fetch of PDFs first then HTML to `/downloads/`, tracked in `downloads.csv` | +| 8 | Zip + download (cell 10) | Bundle all converted `.md` as a single zip for upload to Open WebUI | + +## File + +| File | Purpose | +|---|---| +| `000-configs/notebooks/docx_to_markdown_openwebui.ipynb` | The notebook — 29 cells, nbformat-valid | +| `000-configs/notebooks/README.md` | Index of all notebooks, how to add new ones | + +## How to run (final user workflow) + +1. Open the `.ipynb` in **Colab free tier** → `Runtime → Run all` (or run cell-by-cell). +2. Cell 2: drop the docx/md files (Colab upload picker), or it auto-extracts from `/drive/MyDrive/zzz-new_shit/*.tar`. +3. Cell 5b (idempotent): watch the pre-flight dedup report — see which `(1)`/`(2)` dupes got dropped by content overlap. +4. Cell 6: conversions run; manifest is updated with statuses (`converted` / `crashed_fallback_ok` / `crashed_skipped`). +5. Cell 7: `sources.csv` gets all URLs from the new conversions. +6. Cell 8: heading-structure audit (counts H1/H2/H3 per file — sanity check for Open WebUI chunking). +7. Cell 10: zip + download everything → upload zip into Open WebUI knowledge base. + +## Drive layout (created on first run) + +``` +/drive/MyDrive/to_convert/ +├── manifest.csv ← SHA-256 dedup history (append-only) +├── dedup_drops.csv ← pre-flight fuzzy-dedup log (new today) +├── archive/ ← original input bytes, durable across Colab wipes +├── markdown_output/ ← converted .md files (this is what you upload) +├── sources.csv ← doc → URLs (append-only) +└── downloads/ + ├── pdf/ html/ other/ + └── downloads.csv ← URL → saved file (append-only) +``` + +## Config (cell 9 flags — tunable without touching the converter) + +```python +SKIP_ALREADY_SEEN = True +ARCHIVE_INPUTS = True +DEDUP_SIMILARITY_THRESHOLD = 0.97 # set to 0 to disable fuzzy dedup +DEDUP_CHUNK_SIZE = 500 # chars per chunk in the similarity signature +EXTRACT_IMAGES = True # Extract media to per-doc media/ (no embedding signal yet) +``` + +## Bug fixed this session + +- **TypeError: 'NoneType' object is not subscriptable** in cell 6 when manifest has a hash in the seen-set but `is_seen()` returns None (race with `manifest.csv` written by another process). Guarded with `seen_row = is_seen(sha, manifest) or {}` and `.get(..., '(unknown)')` accessors. + +## Outstanding + +- [ ] Re-run the notebook end-to-end with the user's real upload batch (the 257-file set with the `clean_email_threads.docx` + `clean_email_threads (2).docx` 25-byte-diff pair); confirm the fuzzy dedup actually drops the `(2)` variant at threshold 0.97. +- [ ] If threshold is too generous (drops genuinely different docs), bump to `0.95`. If too strict (lets dupes through), drop to `0.90`. The user has the knob in cell 9 — no code changes needed. +- [ ] Confirm `manifest.csv`, `dedup_drops.csv`, `sources.csv` all appear on Drive after first run and round-trip correctly across Colab sessions. +- [ ] Consider adding a MinHash variant if Jaccard over fixed chunks ever gets slow on >1k inputs. + +## Build provenance + +The notebook is built from a Python script (`/tmp/build_notebook.py`) that stitches fragments together and is then validated. That script is a build artifact and lives outside the vault (single-use); future revisions to the notebook can be made directly by editing the `.ipynb` in Colab and committing the saved file, OR by patching the build script and rerunning. The fragment files were in `/tmp/nbfrags/` at build time — restore from git history if rebuilding. + +## Related + +- [[06-research/gemini-notebook-poweruser-briefing]] — context for the Gemini-Notebook .docx format quirks (cite markers, masthead, bold-as-heading) that this converter handles