What I learned indexing 27,000 ATS feeds

8 September 2026

I got tired of job data being either a scraper that breaks every week or a $2k/month enterprise contract, so I built the boring version. It reads official public ATS feeds on a schedule, normalizes them into one schema, and tracks what changes.

Right now that's about 820,000 open postings from 27,247 boards across Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee, Personio and Workday, in 167 countries. The live numbers are at /v1/stats, which is free and needs no key, so you can check those instead of taking my word for it.

None of the interesting problems were the ones I expected. Parsing seven ATS formats is tedious but it's a known quantity. What took the time was working out what the data means when a feed misbehaves, because almost every wrong answer this thing can give a customer comes from believing a bad read.

An empty feed is not a company that stopped hiring

A board that returns zero postings is almost never a company that fired everyone. It's usually a feed having a bad day. So an empty response never closes anything here. The board gets flagged for review and its postings stay open, because telling a subscriber that 300 roles closed overnight is much worse than telling them nothing changed.

What I got wrong was leaving it there. A board that genuinely emptied out kept its postings open forever, which is exactly as false as closing them on the first blip. The rule needed an ending: if every crawl for a week comes back empty, it isn't a broken read anymore, and the postings close with a reason attached.

Truncation is worse than an error, because it looks like success

CVS Health publishes 19,017 postings. My pagination guard stopped at 10,000. No error, no empty feed, no failed request. Just fewer rows, and a crawl that looks perfectly healthy in every log line and every dashboard.

The damage isn't the missing rows, it's what happens next. If the feed's ordering shifts between crawls, the following crawl reads a different 10,000, and every posting that fell out of the window gets closed. A silent read limit turns into thousands of false "this job is gone" events, which is the single worst thing this product can do.

Two fixes. A read that stopped at the guard now closes nothing, because it isn't a view of the whole board. And the way you find these after the fact is to look for round numbers: a board whose open count is an exact multiple of the platform's page size is a suspect. That query found seven SmartRecruiters boards sitting on exactly 5,000 postings, including one employer publishing 24,689.

A keyset cursor is not enough for a change feed

Everyone knows to paginate with a keyset instead of OFFSET when rows are being written underneath you. That part I had right from the start.

It still wasn't enough. Change events get their ids from a Postgres sequence, one crawl's worth of updates is one transaction, and transactions don't commit in the order they started. So a subscriber can read up to event 1000 while events 990 to 999 are still in flight in a crawl that hasn't committed. Their next cursor is 1000, those nine events are never served to them, and nothing anywhere reports a problem. If one of them was a "closed", the subscriber's copy of the world is quietly wrong forever.

Timestamps have the same hole, because now() in Postgres is the transaction's start time. The fix is to serve only rows written by a transaction older than the oldest one still running, which Postgres will tell you if you ask. Read-only queries never hold it back, so a backup doesn't stall the feed.

A name is not an identity

Greenhouse's payload says "Acme, Inc." The Lever slug for the same employer prettifies to "Acme". Matching companies on the name made those two employers, which meant the company view showed half the roles, and the cross-ATS deduplication couldn't see that two boards belonged to one company.

Matching on a normalized key fixes most of it and then overreaches. Run against the real index, it wanted to merge Zip Co with an unrelated Zip, Method Co. with Method, and Hive with Hive Co. The account slug the company chose on the platform is much better evidence than a display name, and it separates all three.

Some pairs share a name and a slug and are still different companies. Those you settle by reading what each side actually hires for. One ATLAS wanted a Head of Credit and Underwriting. The other wanted an Orthopädieschuhmacher. It's a shoe factory.

What it's for

The endpoint I care about most is /v1/jobs/changes. It returns only what opened, changed, closed or reopened since your cursor, with a field-level diff. Polling a search endpoint to detect change is what most integrations end up doing, and it's both expensive and wrong. This is a cursor you can leave running.

There's also an MCP server, so an agent can query it directly, and an Apify actor. A free key needs no account, no email and no card:

500 records a month free. It's a few days old, so if you try it and something is wrong, I'd genuinely like to know: index@roleset.io.

Back to roleset.io