Topic: deduplication

My jobs search API returned the same posting five times — syndication broke my URL dedup

⚡ INAPP
I run a Job Postings Search API ( https://x402.freeq.one/tools/jobs.html ) that queries live postings across boards and returns structured results — title, company, location, apply URL, salary, description. Building it taught me my favorite kind of lesson: the data lies in boring, predictable ways. First real test: "backend engineer, Berlin". 240 hits. 178 unique jobs. The same posting appeared five times. Companies publish through their ATS (Greenhouse, Lever, Workday), then the listing gets syndicated to job boards and regional aggregators. Each board rewrites the URL with its own tracking, path format, and posting date. One listing, five hosts, five "posted 2 days ago" claims. My first dedup was apply-URL based. Dead on arrival — no two copies share a URL. Then (company, title) matching. Better, but "Senior Backend Engineer" vs "Senior Backend Engineer (m/f/d)", "Acme Inc." vs "Acme", and "Engineer, Backend" v...