A user's job-matching agent kept ranking near-identical listings against each other, burning tokens and crowding out real candidates. I pulled a sample to investigate: one search for "data engineer" in Berlin returned 1,000 results — and 359 of them were duplicates of other entries in the same result set. My dedup key was the URL. That was the bug. Job boards rewrite URLs for tracking. One board appended utm_source , another used gh_src , a third proxied every apply link through a /jobclick?id=... redirector before it ever reached the company page. Three boards, one job, three URLs guaranteed distinct. Exact-match URL dedup caught none of it. The fix ended up being three layers, cheapest first: 1. Canonicalize URLs. Strip query strings, lowercase the host, drop trailing slashes and fragments. This alone removed about half the duplicates. 2. Normalize title + company as the real fingerprint. Lowercase, strip punctuation and bracketed tags like (Remote) , collapse wh...