shit_in_a_vault/07-tasks/docx-to-markdown-converter-notebook.md

title, status, folder, tags, created, updated, version, priority, due, owner

title	status	folder	tags	created	updated	version	priority	due	owner
Task: DOCX→Markdown converter notebook for Open WebUI	active	07-tasks	task notebook openwebui docx markdown converter	2026-07-30	2026-07-30	1.0.0	high	2026-07-30	stateofshit

Task: DOCX→Markdown converter notebook for Open WebUI

Done — 2026-07-30

What

A Colab notebook that batches .docx (and .md) inputs into clean Markdown optimized for embedding in Open WebUI knowledge bases. Open WebUI's RAG uses Markdown Header Splitting, so the converter preserves a clean single-H1 / nested-H2-H3 heading hierarchy — that's what makes retrieval chunks good. Output is mirrored to Google Drive so nothing is lost across Colab wipes.

This notebook is the upgrade over the older /home/user/00-incoming/docx_to_markdown.ipynb: it adds the Gemini-Notebook setup nuance (cite markers stripped, hard bold-promoted to headings) and the dedup edge-cases we hit this week.

Pipeline (per file)

• DOCX: mammoth(html) → clean_markdown → strip_masthead → strip_cite_markers → collapse_hr → promote_structure → collapse_extra_h1 → conditional synthetic H1 prepend
• MD (raw uploaded): same chain minus mammoth
• Fallback: if mammoth crashes (the _accept0 bug on weird docx), the pipeline degrades to a python-docx parser that still produces clean markdown
• Output: one .md per input → /drive/MyDrive/to_convert/markdown_output/

Headlining features

#	Feature	Why it matters
1	Drive-backed manifest dedup (SHA-256 of input bytes)	Renaming a file does NOT cause a reconvert. Survives Colab wipes.
2	Pre-flight fuzzy dedup (new today — cell 5b)	Drops (1)/(2)/Copy of variants where the bytes differ by a few chars but the doc is really the same. Uses Jaccard similarity over 500-char normalized text chunks. Threshold 0.97 (tunable).
3	Gemini masthead + cite-marker stripping	Removes — / Created By: Due Process Demolition / APP: Gemini Notebook / DATE: … boilerplate and [cite: N] artifacts so the KB doesn't fill with garbage
4	Auto heading promotion	For Gemini-Notebook-style docs that use bold paragraphs (not Word heading styles) for section titles, the converter promotes **Bold** lines to real ##/### so Open WebUI's splitter has structure
5	Single-H1 enforcement	collapse_extra_h1() keeps the first H1 (title) and demotes any subsequent H1 → H2. Required by Open WebUI's Markdown Header Splitter
6	URL extraction	Every http(s):// URL found in converted .md files → sources.csv paired with source-doc SHA + stem
7	Bulk downloader (off by default, cell 11)	Once sources.csv is curated, opt-in fetch of PDFs first then HTML to /downloads/, tracked in downloads.csv
8	Zip + download (cell 10)	Bundle all converted .md as a single zip for upload to Open WebUI

File

File	Purpose
000-configs/notebooks/docx_to_markdown_openwebui.ipynb	The notebook — 29 cells, nbformat-valid
000-configs/notebooks/README.md	Index of all notebooks, how to add new ones

How to run (final user workflow)

1. Open the .ipynb in Colab free tier → Runtime → Run all (or run cell-by-cell).
2. Cell 2: drop the docx/md files (Colab upload picker), or it auto-extracts from /drive/MyDrive/zzz-new_shit/*.tar.
3. Cell 5b (idempotent): watch the pre-flight dedup report — see which (1)/(2) dupes got dropped by content overlap.
4. Cell 6: conversions run; manifest is updated with statuses (converted / crashed_fallback_ok / crashed_skipped).
5. Cell 7: sources.csv gets all URLs from the new conversions.
6. Cell 8: heading-structure audit (counts H1/H2/H3 per file — sanity check for Open WebUI chunking).
7. Cell 10: zip + download everything → upload zip into Open WebUI knowledge base.

Drive layout (created on first run)

/drive/MyDrive/to_convert/
├── manifest.csv            ← SHA-256 dedup history (append-only)
├── dedup_drops.csv         ← pre-flight fuzzy-dedup log (new today)
├── archive/                ← original input bytes, durable across Colab wipes
├── markdown_output/        ← converted .md files (this is what you upload)
├── sources.csv             ← doc → URLs (append-only)
└── downloads/
    ├── pdf/  html/  other/
    └── downloads.csv       ← URL → saved file (append-only)

Config (cell 9 flags — tunable without touching the converter)

SKIP_ALREADY_SEEN = True
ARCHIVE_INPUTS    = True
DEDUP_SIMILARITY_THRESHOLD = 0.97   # set to 0 to disable fuzzy dedup
DEDUP_CHUNK_SIZE = 500               # chars per chunk in the similarity signature
EXTRACT_IMAGES = True                # Extract media to per-doc media/ (no embedding signal yet)

Bug fixed this session

• TypeError: 'NoneType' object is not subscriptable in cell 6 when manifest has a hash in the seen-set but is_seen() returns None (race with manifest.csv written by another process). Guarded with seen_row = is_seen(sha, manifest) or {} and .get(..., '(unknown)') accessors.

Outstanding

• Re-run the notebook end-to-end with the user's real upload batch (the 257-file set with the clean_email_threads.docx + clean_email_threads (2).docx 25-byte-diff pair); confirm the fuzzy dedup actually drops the (2) variant at threshold 0.97.
• If threshold is too generous (drops genuinely different docs), bump to 0.95. If too strict (lets dupes through), drop to 0.90. The user has the knob in cell 9 — no code changes needed.
• Confirm manifest.csv, dedup_drops.csv, sources.csv all appear on Drive after first run and round-trip correctly across Colab sessions.
• Consider adding a MinHash variant if Jaccard over fixed chunks ever gets slow on >1k inputs.

Build provenance

The notebook is built from a Python script (/tmp/build_notebook.py) that stitches fragments together and is then validated. That script is a build artifact and lives outside the vault (single-use); future revisions to the notebook can be made directly by editing the .ipynb in Colab and committing the saved file, OR by patching the build script and rerunning. The fragment files were in /tmp/nbfrags/ at build time — restore from git history if rebuilding.

Related

• 06-research/gemini-notebook-poweruser-briefing — context for the Gemini-Notebook .docx format quirks (cite markers, masthead, bold-as-heading) that this converter handles