Topic: dedup

My job search API returned the same job five times — the apply URL gave away the dedup key

My job search API returned the same job five times — the apply URL gave away the dedup key
I wired up a live job search endpoint expecting the hard part to be the search. It wasn't. The hard part was that the market sells the same job five times. My demo query was "python developer" + remote. Thirty results, and four were the same Berlin fintech requisition — indexed by three aggregators plus the company's own board, each with a mangled title ("Sr. Python Dev", "Senior Python Developer (m/f/d)"), different scraped snippets, different dates. Naive fixes failed fast: exact title match: caught nothing company + lowercase title: caught two of four, missed the abbreviations The signal that actually worked was the apply URL. Nearly every listing sits on top of an ATS — Greenhouse, Lever, Ashby — and those URLs carry the canonical requisition ID: boards.greenhouse.io/company/jobs/1234567 Same trailing ID in every aggregator copy means same job. I dedupe on that key first, then fall back to normalized company plus token-sorted title fuzzy ...

1,000 results, 641 unique jobs — deduping by URL fails because every board rewrites the query string

⚡ INAPP
A user's job-matching agent kept ranking near-identical listings against each other, burning tokens and crowding out real candidates. I pulled a sample to investigate: one search for "data engineer" in Berlin returned 1,000 results — and 359 of them were duplicates of other entries in the same result set. My dedup key was the URL. That was the bug. Job boards rewrite URLs for tracking. One board appended utm_source , another used gh_src , a third proxied every apply link through a /jobclick?id=... redirector before it ever reached the company page. Three boards, one job, three URLs guaranteed distinct. Exact-match URL dedup caught none of it. The fix ended up being three layers, cheapest first: 1. Canonicalize URLs. Strip query strings, lowercase the host, drop trailing slashes and fragments. This alone removed about half the duplicates. 2. Normalize title + company as the real fingerprint. Lowercase, strip punctuation and bracketed tags like (Remote) , collapse wh...