Reliability
The product-level semantics are described in Retries, recovery & delivery semantics. This page covers how the platform behaves when its own dependencies fail.
Failure matrix
| Failure | Effect | Recovery |
|---|---|---|
| API replica crashes | In-flight requests fail; clients retry with the same idempotency key | Other replicas serve; container restarts |
| WhatsApp worker crashes | Its sessions drop; in-flight sends become UNKNOWN (or are re-queued if never attempted) | Leases expire after LEASE_TTL_MS; other workers (or the restarted one) claim and restore without QR |
| Worker deploy (SIGTERM) | Sockets closed without logout, leases released | Immediate takeover by remaining/new workers |
| Webhook worker down | Events accumulate in the outbox | Fan-out and delivery resume from the database |
| Scheduler down | Expiry/retention/metrics snapshots pause | Jobs resume; nothing is lost |
| PostgreSQL unavailable | API returns errors for writes (5xx), readiness fails; workers cannot claim or record; heartbeats fail | Self-fencing closes sockets after one TTL; everything resumes from database state when it returns |
| Redis unavailable | Commands and nudges are delayed (workers fall back to polling/claim loop), QR codes cannot be served, rate limiting fails open, BullMQ maintenance pauses | Reconnects automatically; no durable state lives only in Redis |
| S3 unavailable | Media uploads and media sends fail (MEDIA_UNAVAILABLE if it persists through retries) | — |
| WhatsApp unreachable | Sessions go RECONNECTING; messages wait in the queue until reconnect or expiry | Backoff reconnect; ERROR after 12 attempts (manual reconnect) |
| Customer webhook endpoint down | Deliveries retry for ~16 h, then dead | Replay from dashboard/API; /v1/events for reconciliation |
Design rules that make this work
- PostgreSQL is the only source of truth. Redis holds caches, nudges, rate-limit buckets and ephemeral QR codes; losing Redis loses nothing that cannot be rebuilt.
- No dual writes. The message queue and webhook outbox are tables written in the same transaction as the business change.
- Intent in the database, commands as nudges. A lost BullMQ command is corrected by the next reconciliation.
- Fencing everywhere it matters. Session key writes and claims are guarded by
lease_epoch. - Pessimistic about outcomes. Anything that might have been sent is
UNKNOWN, never silently retried. - Bounded everything. Retries (sends: until expiry; webhooks: 8 attempts; reconnects: 12), queue retention in BullMQ, backlog per project, upload sizes, bodies read from customer servers (1 KB preview of webhook responses; media imports capped at the media size limit), HTTP timeouts.
Graceful shutdown
Each process registers ordered shutdown steps (ShutdownManager) with a hard deadline (API: Nest shutdown hooks;
workers: 30–60 s) — stop accepting work, drain, close connections. Compose sets stop_grace_period accordingly.
Health checks
/health/live: the process is running (no dependency checks)./health/ready: PostgreSQL and Redis respond; for workers also that the ownership/dispatch loop ran recently.- A disconnected or restricted WhatsApp account never affects health; instance health is reported through status, events and metrics instead.