Webhooks, queues, and retries: automation reliability 101

Webhooks: the right way to receive events

A webhook is a POST request sent by an external service when something happens. Simple in concept, fragile in practice. The service sending the webhook usually has a timeout — if your endpoint doesn't respond with a 200 within a few seconds, it marks the delivery as failed. If your handler does any real work synchronously — database writes, API calls, AI processing — it will regularly exceed that timeout for legitimate events.

The fix is to decouple receiving from processing. Your webhook endpoint should do exactly three things: validate the request (check the signature if the sender provides one), add the payload to a queue, and return 200 immediately. Everything else happens asynchronously from the queue. This pattern means your webhook endpoint is always fast, always returns 200, and can never be overwhelmed by a burst of events — the queue absorbs the burst and processes it at whatever rate your workers can handle.

Webhook signature validation is a security step that's often skipped. Without it, anyone who knows your webhook URL can send fabricated events. Most services that send webhooks — Stripe, GitHub, Shopify — include a signature header computed from the payload and a shared secret. Verify this signature before processing any payload. In Make.com and n8n, this validation can be added as the first step in the scenario or workflow.

Queues: absorbing bursts and decoupling systems

A queue is a buffer between the thing that generates work and the thing that processes it. Without a queue, your system can only process as much as it can handle simultaneously — peak load determines your infrastructure requirements, and anything above peak fails. With a queue, peak load fills the queue and processing catches up as capacity allows. The tradeoff is latency: queued work is processed later, not immediately.

For automation workflows, the right queue tool depends on your infrastructure. If you're running n8n in queue mode, Redis with Bull is already in your stack. If you're building custom automation with Node.js or Python, Redis with BullMQ or a cloud queue service like AWS SQS or Google Cloud Tasks are solid choices. For low volume, even a Postgres table used as a queue works reliably — select a pending job, mark it as in-progress, process it, mark it as done.

Dead letter queues are the feature of queuing infrastructure that most people don't set up until they've had a production incident. A dead letter queue is where jobs go after they've failed repeatedly and can't be retried. Without it, failed jobs either block the queue or disappear silently. With it, you have a record of everything that failed, why it failed, and when — which makes debugging production incidents significantly faster.

Retries: exponential backoff and idempotency

Most transient failures in automation workflows are caused by temporary conditions: a downstream API is overloaded, a database connection timed out, a rate limit was hit. These failures resolve themselves. Retrying after a short delay succeeds the vast majority of the time. But retrying immediately, as fast as possible, makes the problem worse — an overloaded service hit by immediate retries stays overloaded.

Exponential backoff is the standard solution: retry after 1 second, then 2, then 4, then 8, then 16. The delay grows exponentially, giving the failing service time to recover before each retry. Add jitter — a small random variation on each delay — to prevent multiple workers from retrying at exactly the same time and overwhelming the service in waves. Most queue libraries support exponential backoff with jitter natively.

Retries require idempotency: the ability to process the same event multiple times without causing duplicate effects. If retrying a job sends the same email twice or processes the same payment twice, retries are worse than failures. Design every handler to be idempotent. Check whether the action has already been performed before performing it. Use a unique identifier from the original event as an idempotency key when calling external APIs that support it.

• Attempt 1: immediate
• Attempt 2: 1s + jitter
• Attempt 3: 2s + jitter
• Attempt 4: 4s + jitter
• Attempt 5: 8s + jitter
• After max retries: move to dead letter queue, alert on-call

Monitoring: catching failures before users do

Silent failures are the most dangerous kind. A workflow that fails with an error you can see is fixable. A workflow that appears to succeed but produces wrong output, or a workflow that stops processing events without returning any error, can run silently broken for days. The only way to catch these is monitoring.

For event-driven workflows, the most important metric is processing lag — the time between an event arriving and it being processed. If lag is growing, either processing is slower than the event arrival rate, or jobs are failing and retrying. Alert when lag exceeds your SLA. For scheduled workflows, alert when a workflow doesn't start within a reasonable window of its scheduled time — a scheduled workflow that silently doesn't run is a common failure mode.

Build a heartbeat mechanism for every critical automation. The heartbeat is a separate scheduled workflow that runs every N minutes and checks whether the primary workflow is producing expected output. If the primary workflow is supposed to process new orders and the heartbeat checks that at least one order was processed in the last hour and no order has been pending for more than 30 minutes, the heartbeat catches both the 'nothing is being processed' failure and the 'processing is too slow' failure.