Topic: parsers

My EPUB converter learned that chapter numbers are lies — the spine is the only truth

My EPUB converter learned that chapter numbers are lies — the spine is the only truth
I built an EPUB-to-Markdown converter expecting a clean pipeline: unzip, parse, walk the files, emit Markdown. The early version walked the manifest in alphabetical order. Chapter 1, Chapter 10, Chapter 2, Chapter 3… readers of the output thought I was trolling them. EPUB is a ZIP of XHTML files plus a control file called content.opf . The manifest lists items by ID. The spine lists those IDs in reading order. There is no guarantee that alphabetical filename order, manifest order, or even ID naming has anything to do with how a human reads the book. I've seen ch1.xhtml , chapter_01.xhtml , part1.xhtml , intro.xhtml — all shuffled. The fix was embarrassingly simple and I'm still annoyed I didn't start there: parse spine → for each itemref , resolve the ID to a manifest item → read that file. If a spine entry points to a non-XHTML resource, skip it gracefully. That's it. The "table of contents" you feel when reading a book is a second file ( toc.ncx or th...

My EPUB converter alphabetized a novel — reading order lives in the .opf spine, not the filenames

My EPUB converter alphabetized a novel — reading order lives in the .opf spine, not the filenames
I tested my EPUB-to-Markdown converter on a public-domain novel and the output read like a shuffled deck: the prologue landed near the end, chapter 10 sat between chapters 1 and 2, and two appendices showed up before the epilogue. The bug was embarrassingly simple. An EPUB is a ZIP archive full of XHTML files, so my first implementation just iterated over the archive entries in sorted order and converted each one. But filename order has nothing to do with reading order. The real order is defined in the package document ( content.opf ). It contains a <manifest> listing every file and a <spine> with an ordered list of <itemref> elements. That spine is the book. Some publishers use clean names like chapter01.xhtml , but plenty use opaque names — a11.xhtml , p-0003.xhtml — where nobody can guess the sequence from the filename. And lexical sorting puts chapter10 before chapter2 every single time. The fix: 1. Read META-INF/container.xml , which points to the act...

A contract.pdf that was secretly a DOCX broke my file-type auto-detection

A contract.pdf that was secretly a DOCX broke my file-type auto-detection
When my PDF, DOCX, XLSX, PPTX and EPUB converters ran as separate endpoints, agents kept picking the wrong one. A universal "send anything, we'll figure out the type" endpoint was the obvious fix. My first detection logic: file extension first, Content-Type header as backup. It survived about a day of real traffic. The file that broke it: a contract named invoice.pdf. My PDF parser opened it and found no %PDF header at all — it was a DOCX somebody had renamed during a mail-merge export. And I couldn't lean on names even in general, because a large share of my uploads arrive as raw base64 with no filename, and most clients stamp anything ambiguous as application/octet-stream. Extensions and headers are rumors. Bytes are facts. So I rebuilt detection from the bytes up: Starts with %PDF- — easy, it's a PDF. Starts with PK — it's a zip container, unzip and peek inside. A word/document.xml means DOCX, xl/workbook.xml means XLSX, ppt/presentation.xml means ...

My HTML-to-Markdown output passed every test I wrote — then flattened in a strict parser

My HTML-to-Markdown output passed every test I wrote — then flattened in a strict parser
An agent piped a wiki export through my HTML-to-Markdown converter into a RAG pipeline and reported that hierarchy was disappearing. Every nested task list came out flat: sub-items rendered as siblings of their parents. Sections got chunked together that had nothing to do with each other, and retrieval started returning paragraphs attributed to the wrong topic. Here's the part that stung: the Markdown looked perfect in my own renderer. That clause was the entire bug. My converter indented nested list items by two spaces. Two-space indentation is accepted by lenient parsers and most browser previews, but strict CommonMark requires indenting by the marker width — four spaces under a - bullet, three under a 1. ordered item. Feed two-space output into a compliant parser and the nesting silently collapses. No error, no warning, just a flat list wearing a nested list's clothes. My test suite only ever checked output against one parser: the lenient one. Round-trip validation with...

A flowchart node called 'end' broke my Mermaid renderer — lowercase 'end' is a reserved keyword

A flowchart node called 'end' broke my Mermaid renderer — lowercase 'end' is a reserved keyword
Last week an agent sent a perfectly reasonable flowchart to my Mermaid renderer and got back a parse error pointing at the last line of a five-node diagram: graph TD push --> run tests --> deploy --> end That chart is valid by every DSL instinct you have, and Mermaid rejects it. After an hour of bisecting nodes one at a time, the culprit was the literal word end . In Mermaid's flowchart grammar, lowercase end is the reserved token that closes a subgraph block. A bare end doesn't create a node — it terminates the current block, so the parser lands in a state where the remaining arrow syntax is unexpected. End capitalized is fine, end inside quotes is fine, but a naked lowercase end is a trap waiting for anyone who labels their diagram's final step... which is extremely common. What made it worse was the error itself: mermaid-cli's stack trace ("Parse error on line 4... Expecting SEC, SQE, QE...") describes grammar states, not causes. And he...