API Pollers: For the APIs That Never Learned to Send Webhooks

The APIs That Never Got the Memo

Webhooks assume the other side wants to push. Plenty of APIs don't work that way — internal tools, legacy vendor integrations, and a long tail of SaaS products that only ever shipped a GET /items endpoint and called it done. If you wanted events out of one of those, your options were "write and host a poller yourself" or "don't."

API Pollers are that poller, hosted, scheduled, and wired straight into the pipeline you already have. Point one at a REST endpoint, tell it where the array of items lives in the response and which field identifies an item, and it starts turning new and changed rows into inbound events on a source you already own — no separate integration path, no second set of routes and transforms to maintain.

What's New

• Schedule a poll on a cron expression, with human presets (every 15 minutes, hourly, daily) or raw cron for anything more specific, per-poller timezone included
• Point it at any REST endpoint — custom headers for auth, an optional response_path (e.g. data.items) to pull the array out of a wrapped response, an id_field to identify each item, and an optional event_type_field to pick the emitted event's type from the item itself (falling back to a default you set)
• Only new or changed items get emitted — a rolling window of item keys and content hashes travels with the poller, so a poll that returns 500 unchanged rows emits nothing
• Runs through the real ingest pipeline — every emitted item is a genuine POST to the bound source's own /ingest/:orgSlug/:sourceSlug URL, signed the same way a real webhook would be, so your existing routes, filters, transforms, and dedup all apply exactly as they do to inbound traffic from anywhere else
• "Trigger now" for testing a poller's configuration without waiting for its schedule
• Plan-gated, and wired to run automatically on your configured schedule — no separate service to babysit

Poll, Diff, Emit — and Nothing Else Special

The core loop is deliberately small: fetch the URL, pull the item array out (directly, or via response_path if the response wraps it), and for each item compute a key from id_field and a hash of the whole item. Compare both against a seen_items window carried on the poller row — new key, it's a new item; same key with a different hash, it changed; same key and same hash, skip it. Everything that's new or changed gets POSTed to the source's ingest URL as { type: eventType, data: item }, HMAC-signed with the source's own signing secret exactly like the provider on the other end of a real webhook would do it.

That last part is the design decision that matters most: a poller has no special code path into your pipeline. It doesn't write to your event history directly, and it doesn't get a shortcut around routes or transforms. It calls the exact same public ingest endpoint a real webhook provider would call, with a real signature, and from there it's indistinguishable from any other inbound event — same filters decide whether it routes anywhere, same transforms reshape it, same dedup and schema validation apply. Building a poller doesn't mean re-learning a second, poller-specific configuration surface for what happens after ingestion — it's just another way an event gets in the door.

The seen-items window is intentionally a rolling, capped structure rather than a record of every item ever seen — capped at 500 entries per poller so a large or badly-behaved upstream API can't grow it without bound. An item that scrolls out of the window and later reappears in a poll gets treated as new again. That's a real tradeoff, not an oversight: for the steady, moderate-volume feeds pollers are built for, it's the difference that keeps this a small feature instead of one that needs its own storage and cleanup story.

Concurrency Without Double-Firing

Pollers run off a scheduled tick, and the tick doesn't wait to see whether the last run finished. Each run claims its due pollers with a locking strategy that skips anything already claimed, and immediately pushes each claimed poller's next run time out by a five-minute lease before doing any fetch or emit work. Two ticks landing close together can't both grab the same poller — the second one just moves past a poller already claimed. And if a run crashes mid-fetch before it gets to persist a real result, the lease expires on its own and the poller becomes eligible again rather than being stuck waiting on a run that's never coming back.

Try It

Open Inbound → API Pollers and create one: pick a source to bind it to, the endpoint to poll, and a schedule (a preset or your own cron expression). Fill in response_path if the array is nested, set id_field to whatever uniquely identifies a row, and hit Trigger now to see the first poll happen immediately rather than waiting for the schedule. From there, treat the emitted events like any other inbound traffic on that source — because that's exactly what they are.