RELIABILITY PLATFORM · WEBHOOKS

Relay

A webhook delivery platform that accepts an event, commits its fan-out, and recovers deliveries without repeating the original business action.

RoleDesign & implementation
TypeIndependent project
StackNode.js, MongoDB (replica set), Supabase Auth

The problem

Context

Webhooks look like a single HTTP call. In production they're a distributed systems problem: the sender crashes mid-request, the receiver times out after it already processed the event, a destination goes down for an hour, and a duplicate submission arrives from a client retry. Relay is built around one question: once an event is accepted, how do you guarantee every destination eventually gets it, without ever re-running the action that produced it?

Constraints

Durable before responsive.

  • Ingress must be durable before it responds. A 202 has to mean the event and every fan-out delivery exist.
  • Client retries must not create duplicate events.
  • A crashed worker's work must be recoverable by another worker, and a slow zombie worker must not overwrite its replacement.
  • Receivers must be able to verify a request came from Relay and wasn't replayed.
  • Customer-supplied URLs must not become a path into private networks (SSRF).
  • Workspaces must be fully isolated from one another.
  • Run on modest infrastructure: one existing MongoDB Atlas replica set, no extra broker.

Architecture

Transactional fan-out, recoverable delivery.

CLIENT / OPS UINATIVE HTTP APIMONGODBDESTINATIONWORKER / 60S LEASERETRY / DLQ

API and worker can run in one process for development or as separate processes against the same database.

Key decisions

Designing the delivery contract.

Decision 1

Return 202 only after the transaction commits

Alternatives considered

Respond immediately and enqueue asynchronously; hold the request open until delivery succeeds.

Why

One MongoDB transaction writes the ingress receipt, every fan-out delivery and the audit record. If it commits, the work exists; if it doesn't, nothing does.

Decision 2

Use MongoDB as the queue instead of a broker

Alternatives considered

BullMQ + Redis; QStash.

Why

Keeping the queue in the same transactional database as the ingress receipt removes the database-to-broker dual-write gap entirely. At higher volume, the answer is a broker fed by a transactional outbox, not a dual write.

Decision 3

Leases with fencing tokens instead of locks

Alternatives considered

Long-held locks; trusting one worker per job.

Why

Workers reserve a delivery with an atomic compare-and-set and a 60-second lease. Completion is fenced by the lease token, so a stale worker cannot overwrite its replacement's result.

Decision 4

Be honest about at-least-once delivery

Alternatives considered

Claim exactly-once.

Why

Relay can't prove a receiver did not accept a request before a worker crashed, so it doesn't pretend to. Every request carries a stable relay-event-id for the receiver to deduplicate on.

Decision 5

Idempotency keys with content fingerprints

Why

A unique workspace + idempotency-key index resolves concurrent duplicates at the database level. Reusing a key with a different payload returns 409. The fingerprint uses canonical JSON, so key order doesn't matter.

Failure modes & recovery

Failure is a first-class path.

What goes wrongWhat the system does
Client retries the same submissionUnique index returns the original receipt; different content with the same key → 409.
Worker crashes mid-deliveryLease expires after 60s; another worker reserves it; the attempt counts against the budget.
Stale worker finishes lateWrite rejected: the lease token no longer matches.
Receiver times out or returns 5xxRetry with the route's policy: 1–10 attempts, fixed or exponential backoff, 100–30,000 ms timeout.
Attempts exhaustedDelivery is dead-lettered with full attempt evidence; an operator can replay it.
Destination resolves to a private IPRejected; DNS is checked on creation and on every attempt, the validated IP is pinned to the connection, and redirects aren't followed.
Receiver sees a replayed requestSignature covers timestamp + raw body; receivers enforce a timestamp window and dedupe on relay-event-id.
  • Signature: v1=<HMAC-SHA256> over timestamp + exact raw body, fresh on each attempt.
  • Route secrets encrypted with AES-256-GCM using the route ID as authenticated context, never returned by the API or shown in the UI.
  • Supabase tokens are verified against Supabase; browser sessions use HttpOnly, SameSite cookies.
  • Request bodies are capped at 100 KB; rate limits are stored in MongoDB so they hold across processes.

Boundaries

What Relay deliberately does not do.

  • At-least-once, never exactly-once.
  • Polling worker; a sleeping free-tier host delays retries, though queued work survives.
  • No retention/archival policy, master-key rotation, machine API credentials or alert integrations yet.
  • Up to 50 routes per workspace and 20 replay cycles, to keep fan-out bounded.

What I'd change next

Next evolution.

  • Move to a dedicated broker fed by a transactional outbox once volume justifies it.
  • Run workers as an always-on separate service so retries aren't delayed by host sleep.
  • Add retention and archival for delivery evidence, plus master-key rotation.
  • Add alerting on dead-letter growth and route health.