stateofshit
7f189254c0
- 000-configs/notebooks/docx_to_markdown_openwebui.ipynb (29 cells) - mammoth + python-docx fallback, manifest dedup (SHA-256), pre-flight fuzzy dedup (Jaccard @ 0.97), URL extraction, gzip download - 000-configs/notebooks/README.md: index of notebooks + how-to-add - 000-configs/README.md: list notebooks/ subfolder in the table - 07-tasks/docx-to-markdown-converter-notebook.md: full task writeup
103 lines
6.5 KiB
Markdown
103 lines
6.5 KiB
Markdown
---
|
|
title: "Task: DOCX→Markdown converter notebook for Open WebUI"
|
|
status: "active"
|
|
folder: "07-tasks"
|
|
tags: [task, notebook, openwebui, docx, markdown, converter]
|
|
created: "2026-07-30"
|
|
updated: "2026-07-30"
|
|
version: "1.0.0"
|
|
priority: "high"
|
|
due: "2026-07-30"
|
|
owner: "stateofshit"
|
|
---
|
|
|
|
# Task: DOCX→Markdown converter notebook for Open WebUI
|
|
|
|
## Done — 2026-07-30
|
|
|
|
## What
|
|
|
|
A Colab notebook that batches `.docx` (and `.md`) inputs into clean Markdown optimized for embedding in **Open WebUI** knowledge bases. Open WebUI's RAG uses Markdown Header Splitting, so the converter preserves a clean single-H1 / nested-H2-H3 heading hierarchy — that's what makes retrieval chunks good. Output is mirrored to Google Drive so nothing is lost across Colab wipes.
|
|
|
|
This notebook is the upgrade over the older `/home/user/00-incoming/docx_to_markdown.ipynb`: it adds the Gemini-Notebook setup nuance (cite markers stripped, hard bold-promoted to headings) and the dedup edge-cases we hit this week.
|
|
|
|
## Pipeline (per file)
|
|
|
|
- **DOCX**: `mammoth(html) → clean_markdown → strip_masthead → strip_cite_markers → collapse_hr → promote_structure → collapse_extra_h1 → conditional synthetic H1 prepend`
|
|
- **MD** (raw uploaded): same chain minus mammoth
|
|
- **Fallback**: if mammoth crashes (the `_accept0` bug on weird docx), the pipeline degrades to a `python-docx` parser that still produces clean markdown
|
|
- **Output**: one `.md` per input → `/drive/MyDrive/to_convert/markdown_output/`
|
|
|
|
## Headlining features
|
|
|
|
| # | Feature | Why it matters |
|
|
|---|---|---|
|
|
| 1 | Drive-backed manifest dedup (SHA-256 of input bytes) | Renaming a file does NOT cause a reconvert. Survives Colab wipes. |
|
|
| 2 | **Pre-flight fuzzy dedup** (new today — cell 5b) | Drops `(1)`/`(2)`/`Copy of` variants where the bytes differ by a few chars but the doc is really the same. Uses Jaccard similarity over 500-char normalized text chunks. Threshold `0.97` (tunable). |
|
|
| 3 | Gemini masthead + cite-marker stripping | Removes `— / Created By: Due Process Demolition / APP: Gemini Notebook / DATE: …` boilerplate and `[cite: N]` artifacts so the KB doesn't fill with garbage |
|
|
| 4 | Auto heading promotion | For Gemini-Notebook-style docs that use bold paragraphs (not Word heading styles) for section titles, the converter promotes `**Bold**` lines to real `##`/`###` so Open WebUI's splitter has structure |
|
|
| 5 | Single-H1 enforcement | `collapse_extra_h1()` keeps the first H1 (title) and demotes any subsequent H1 → H2. Required by Open WebUI's Markdown Header Splitter |
|
|
| 6 | URL extraction | Every `http(s)://` URL found in converted .md files → `sources.csv` paired with source-doc SHA + stem |
|
|
| 7 | Bulk downloader (off by default, cell 11) | Once sources.csv is curated, opt-in fetch of PDFs first then HTML to `/downloads/`, tracked in `downloads.csv` |
|
|
| 8 | Zip + download (cell 10) | Bundle all converted `.md` as a single zip for upload to Open WebUI |
|
|
|
|
## File
|
|
|
|
| File | Purpose |
|
|
|---|---|
|
|
| `000-configs/notebooks/docx_to_markdown_openwebui.ipynb` | The notebook — 29 cells, nbformat-valid |
|
|
| `000-configs/notebooks/README.md` | Index of all notebooks, how to add new ones |
|
|
|
|
## How to run (final user workflow)
|
|
|
|
1. Open the `.ipynb` in **Colab free tier** → `Runtime → Run all` (or run cell-by-cell).
|
|
2. Cell 2: drop the docx/md files (Colab upload picker), or it auto-extracts from `/drive/MyDrive/zzz-new_shit/*.tar`.
|
|
3. Cell 5b (idempotent): watch the pre-flight dedup report — see which `(1)`/`(2)` dupes got dropped by content overlap.
|
|
4. Cell 6: conversions run; manifest is updated with statuses (`converted` / `crashed_fallback_ok` / `crashed_skipped`).
|
|
5. Cell 7: `sources.csv` gets all URLs from the new conversions.
|
|
6. Cell 8: heading-structure audit (counts H1/H2/H3 per file — sanity check for Open WebUI chunking).
|
|
7. Cell 10: zip + download everything → upload zip into Open WebUI knowledge base.
|
|
|
|
## Drive layout (created on first run)
|
|
|
|
```
|
|
/drive/MyDrive/to_convert/
|
|
├── manifest.csv ← SHA-256 dedup history (append-only)
|
|
├── dedup_drops.csv ← pre-flight fuzzy-dedup log (new today)
|
|
├── archive/ ← original input bytes, durable across Colab wipes
|
|
├── markdown_output/ ← converted .md files (this is what you upload)
|
|
├── sources.csv ← doc → URLs (append-only)
|
|
└── downloads/
|
|
├── pdf/ html/ other/
|
|
└── downloads.csv ← URL → saved file (append-only)
|
|
```
|
|
|
|
## Config (cell 9 flags — tunable without touching the converter)
|
|
|
|
```python
|
|
SKIP_ALREADY_SEEN = True
|
|
ARCHIVE_INPUTS = True
|
|
DEDUP_SIMILARITY_THRESHOLD = 0.97 # set to 0 to disable fuzzy dedup
|
|
DEDUP_CHUNK_SIZE = 500 # chars per chunk in the similarity signature
|
|
EXTRACT_IMAGES = True # Extract media to per-doc media/ (no embedding signal yet)
|
|
```
|
|
|
|
## Bug fixed this session
|
|
|
|
- **TypeError: 'NoneType' object is not subscriptable** in cell 6 when manifest has a hash in the seen-set but `is_seen()` returns None (race with `manifest.csv` written by another process). Guarded with `seen_row = is_seen(sha, manifest) or {}` and `.get(..., '(unknown)')` accessors.
|
|
|
|
## Outstanding
|
|
|
|
- [ ] Re-run the notebook end-to-end with the user's real upload batch (the 257-file set with the `clean_email_threads.docx` + `clean_email_threads (2).docx` 25-byte-diff pair); confirm the fuzzy dedup actually drops the `(2)` variant at threshold 0.97.
|
|
- [ ] If threshold is too generous (drops genuinely different docs), bump to `0.95`. If too strict (lets dupes through), drop to `0.90`. The user has the knob in cell 9 — no code changes needed.
|
|
- [ ] Confirm `manifest.csv`, `dedup_drops.csv`, `sources.csv` all appear on Drive after first run and round-trip correctly across Colab sessions.
|
|
- [ ] Consider adding a MinHash variant if Jaccard over fixed chunks ever gets slow on >1k inputs.
|
|
|
|
## Build provenance
|
|
|
|
The notebook is built from a Python script (`/tmp/build_notebook.py`) that stitches fragments together and is then validated. That script is a build artifact and lives outside the vault (single-use); future revisions to the notebook can be made directly by editing the `.ipynb` in Colab and committing the saved file, OR by patching the build script and rerunning. The fragment files were in `/tmp/nbfrags/` at build time — restore from git history if rebuilding.
|
|
|
|
## Related
|
|
|
|
- [[06-research/gemini-notebook-poweruser-briefing]] — context for the Gemini-Notebook .docx format quirks (cite markers, masthead, bold-as-heading) that this converter handles
|