Advanced Delivery Throttling: Rate or Concurrency, Per Destination

A Number Was Never Going to Be Enough

Destinations have had a rate limit for a while: one field, requests per minute, optional. It covered the simple case — "don't send this destination more than 100 requests a minute" — and nothing else. It couldn't express "no more than 3 requests in flight at once" for a downstream service that cares about concurrency, not throughput. It couldn't tell you what was actually happening to a delivery while it waited — the number just capped how fast deliveries left, silently, with no visibility into whether anything was being held back right now.

Throttling is now a real per-destination admission-control system: choose rate-based or concurrency-based limiting, set an optional queue depth so a backed-up destination fails fast instead of piling up deliveries forever, and watch it work in real time on the diagram.

What's New

• Two throttle modes per destination — rate-based (N requests per second/minute/hour) or concurrency-based (at most N deliveries in flight at once)
• Optional queue depth limit — deliveries queued beyond the configured depth fail fast with a clear reason instead of waiting indefinitely
• Live visualization on the Architecture Diagram — a destination under active throttling gets an amber pulse ring, a compact badge showing its configured limit, and delivery dots that visibly stall and fade instead of completing their trip along the edge
• A durable, ordered queue backing it — throttled deliveries wait in strict oldest-first order, backed by real per-destination storage, not a fire-and-hope retry
• Full replacement of the old field — rateLimitPerMinute is gone from the API and database; throttle: { mode, rateLimit, rateUnit, maxConcurrency, queueLimit } is the only way to configure it now

Rate Mode vs. Concurrency Mode

Rate mode is the old rate limiter's job, just more flexible — a request budget (rateLimit) over a window you choose (rateUnit: per second, minute, or hour), enforced with a fixed-window counter per destination.

Concurrency mode is new: instead of capping how often you can send, it caps how many deliveries can be in flight to this destination at the same time. Set it to 1 and every delivery to that destination goes out strictly sequentially — the next one doesn't start until the current one finishes. That matters for destinations where request ordering or backend contention is the actual constraint, not raw throughput — a downstream service that serializes writes internally, for instance, where two simultaneous requests just queue up on their end anyway and you'd rather hold the line on your end where you have visibility into it.

Either mode can carry an optional queue depth limit. Without one, a destination that's throttled harder than its actual delivery volume will just accumulate a growing backlog of waiting deliveries. With one set, once the queue is full, new deliveries fail immediately with a clear queue_full reason instead of waiting behind an unbounded line — the same backpressure principle as any bounded channel, applied to outbound delivery.

A Real Queue, Not a Counter

The old rate limiter kept a single counter — enough for "how many requests so far this window," not enough for "what's actually waiting, in what order." Throttling is now backed by a dedicated, durable queue per destination, with its own persistent state tracking exactly what's waiting and in what order.

That's what makes "oldest deliveries go first" an actual guarantee instead of a best-effort ordering: every throttled delivery is a row, admission is decided against real queue state, and a freshly-arriving delivery is only allowed to jump ahead of a nonempty queue if the queue is empty — never past deliveries already waiting. When a slot frees up (a lease is released), the drain loop grants the next waiting row immediately rather than waiting for the next scheduled tick, so a burst of capacity gets used right away instead of sitting idle until the next timer fires.

Watching It Happen

The most immediate way to see throttling working is on the Architecture Diagram. A destination with throttling configured shows a small badge summarizing its limit — 100/min, ≤3 concurrent — right on the node. When a delivery is actively being held back by that limit, the node gets an amber pulse ring instead of the usual green (success) or red (failure), and the traveling dot representing that delivery visibly behaves differently on the edge: instead of completing its animation from source to destination, it moves about 40% of the way along, wobbles in place like it's stalled, and fades out — because it is stalled, not delivered and not failed, just waiting its turn.

It's a small piece of UI, but it turns "is my rate limit actually doing anything?" from a question you'd have to answer by cross-referencing delivery logs into something you can just watch happen.

Try It

Open any destination's edit form and look for the new Throttling section, where the old single rate-limit field used to be. Pick a mode, set your limit, optionally cap the queue depth, save — then open the Architecture Diagram and send some traffic through it. If the limit is doing anything, you'll see the amber pulse.