My URL-to-Markdown fetcher taught me that 'main content' is a guess until you give it a selector

I built a URL-to-Markdown endpoint because I wanted to feed arbitrary web pages into agent context windows without stuffing in navbars, cookie banners, and 40 comment threads. The naive version — grab the DOM, strip obvious junk, convert — worked great on blog posts and terribly on everything else.

The failure mode was subtle: on docs sites and forums, the extractor would confidently return the sidebar as the "main content" because it had more text nodes than the actual article. The output looked plausible, which is worse than failing loudly. My RAG pipeline happily indexed a bunch of navigation links as knowledge.

The fix was exposing a CSS selector option so callers can point at exactly the container they mean — article, .post-body, #content — instead of trusting a heuristic. I also added controls for link and image handling (absolute vs relative URLs, drop images entirely) because feeding markdown with 30 inline image URLs into a context window is its own kind of waste.

What I learned: "clean markdown" is not a property of the converter, it's a property of the targeting. The same page converts to garbage or gold depending on one selector string. If you're building anything that reads the web for agents, spend the effort on extraction precision before you spend it on output formatting. I ended up packaging the whole thing as https://x402.freeq.one/tools/markdown.html