Stack: Node.js · TypeScript · PostgreSQL · BullMQ · Redis
Use Case: Food Delivery App (think Swiggy / Zomato)
What you'll learn: Why the obvious approach breaks, what replaces it, and how real apps handle it at scale.

The Problem Nobody Talks About

You open Swiggy, tap "Place Order," and within a second you see:

✅ Order placed! Raj's Kitchen is preparing your Paneer Burger.

What just happened in the background? Your order was saved to a database — obvious. But simultaneously:

• You got a push notification + email confirmation

• Raj's Kitchen got an alert to start cooking

• A nearby rider got a pickup request

Three different people. Three different channels. All triggered by one button tap.

And you never waited for any of it.

That's a notification system. And building one that feels instant while reliably delivering to everyone that's the engineering problem.

Let's build it. Twice. First the wrong way, then the right way.

The Cast of Characters

In our food delivery app, there are three actors:

One event — order.placed — needs to reach different people with different messages on different channels.

🤔 Stop and think: At the moment a customer places an order, a rider doesn't exist in the picture yet. When does the rider enter the flow?

Part 1 — The Naïve Version

The Mental Model

The simplest thing you can do: when an order is placed, send all notifications before responding to the user.

Simple. Obvious. Works for 10 users. Completely wrong at scale.

Let's build it anyway — because you need to feel why it's wrong.

The Data Model

Before writing any logic, let's design the tables.

users                   → id, name, role (customer | restaurant | rider)
channels                → id, user_id, type (email | push), endpoint, is_active
notifications           → id, user_id, event_type, created_at
notification_deliveries → id, notification_id, channel_id, status, attempt_count, sent_at, provider_message_id

🤔 Stop and think: Why are channels and notifications separate tables? What's the difference in their lifecycle?

The key insight: for each notification, you need one row per channel in notification_deliveries. The transaction model creates the full hierarchy upfront:

Order
  ↓
Notification (id=42, user=Ayush, event=order.placed)
  ↓
Notification Deliveries
  ├── Email (channel_id=1, status=sent, attempt_count=1)
  └── Push  (channel_id=2, status=pending, attempt_count=0)  ← can retry independently

The Code

// POST /order — the naïve handler
app.post("/order", async (_req, res) => {
  const start = Date.now();

  // Step 1: save order
  const order = { id: 101, item: "Paneer Burger" };

  // Step 2: notify everyone — SYNCHRONOUSLY
  await notify(1, "order.placed");     // customer (email + push)
  await notify(2, "order.placed");     // restaurant (email)

  // Step 3: respond — but only after all notifications sent
  res.json({ success: true, duration_ms: Date.now() - start });
});

And the notification manager:

async function notify(userId: number, eventType: string) {
  const userChannels = channels.filter(c => c.user_id === userId && c.is_active);

  for (const channel of userChannels) {
    await sendOnChannel(channel);  // ← blocking. one at a time.
  }
}

The Sequence — V1

The Load Test Results

We simulated a degraded email provider (5s per send) with 5 concurrent users:

user #1 waited → 10.01s
user #2 waited → 10.01s
user #3 waited → 10.01s
user #4 waited → 10.01s
user #5 waited → 10.01s

💀 10 seconds to hear "your order is placed"

🤔 Stop and think: We ran 10 concurrent users and all got responses in ~3s (not 30s). Why didn't they queue up? And why is that still a problem?

Part 2 — V2: What Breaks in V1 (And Why)

The Root Cause

Everything wrong with v1 has one root cause:

The notification process lives inside the HTTP request lifecycle.

The user is held hostage until every notification is sent. The notification is not a part of placing an order — it's a reaction to it. Reactions should not block the action.

Failure Modes

🤔 Stop and think: If you had to fix v1 tomorrow with minimal changes, what's the one thing you'd change first?

The obvious first fix: don't wait for notifications inside the request. Push a job to a queue, respond immediately.

POST /order
  → Save order to Postgres
  → queue.add({ userId, eventType })   ← fire and forget
  → 200 OK immediately ✅

Response time drops from 3000ms to milliseconds. Problem solved?

Not quite.

The Dual-Write Problem

Postgres and Redis are two independent systems. When you do both sequentially, they are not atomic:

INSERT INTO orders ...    ← Postgres commit ✅
queue.add(job)            ← Redis write

Two failure scenarios:

This is the dual-write problem — you can't have atomicity across two independent systems without a coordination mechanism.

The fix: make Postgres the single source of truth. Write the order and the notification intent to Postgres in one transaction. Then relay to Redis. That's what the Outbox Pattern solves.

Part 3 — The Scaled Version

The New Mental Model

User taps "Place Order"
  → Save order to DB   ┐
  → Write to outbox    ┘  one atomic transaction
  → ✅ Respond: "Order placed!" (< 50ms)

  Meanwhile, in the background...
  → Relay picks up outbox rows → pushes to queue
  → Worker processes each job → sends notifications
  → Retries automatically on failure

The user is gone before a single notification is sent. And that's correct.

The New Architecture

The Outbox Pattern — A Real-World Analogy

[Visual: A sticky note board (the outbox) next to a post box. Person writes note → sticks it on board → postman collects from board → delivers. Even if postman is on leave, notes pile up safely on the board]

Imagine you run a restaurant. You get an order. You write it on a slip and put it in the kitchen window (the outbox). The kitchen picks it up when ready. If the kitchen is slammed, slips pile up in the window safely — they don't disappear.

The outbox pattern works the same way:

-- Everything below is ONE atomic transaction.
-- Either all of it commits, or none of it does.
-- Redis can be completely down — the intent is safe in Postgres.

BEGIN;

  -- Step 1: Save the order (the real business action)
  INSERT INTO orders (customer_id, restaurant_id, item)
  VALUES (1, 2, 'Paneer Burger')
  RETURNING id;  -- → order_id = 101

  -- Step 2: Create the parent notification record per recipient
  -- One row per person who needs to be notified
  INSERT INTO notifications (user_id, event_type, status)
  VALUES
    (1, 'order.placed', 'created'),  -- customer (Ayush)
    (2, 'order.placed', 'created');  -- restaurant (Raj Kitchen)
  -- RETURNING id → notification_id = 42, 43

  -- Step 3: Write to outbox — one row per notification
  -- status = 'pending' means: relay hasn't picked this up yet
  -- This is the "sticky note on the kitchen window" — safely stored,
  -- will be processed when the relay/worker is ready
  INSERT INTO outbox (notification_id, user_id, event_type, status)
  VALUES
    (42, 1, 'order.placed', 'pending'),  -- for Ayush
    (43, 2, 'order.placed', 'pending');  -- for Raj Kitchen

COMMIT;
-- ✅ Transaction done. User gets their response immediately.
-- Relay will poll outbox, find these PENDING rows, and push to BullMQ.
-- Even if Redis is down right now — these rows survive. Nothing is lost.

🤔 Stop and think: The relay marks outbox rows as QUEUED only after queue.add() succeeds. Why not before?

Idempotency — The Safety Net

🤔 Stop and think: Here's the race condition: relay enqueues a job, then marks it QUEUED. But between those two steps, the relay crashes. On restart, the row is still PENDING. Relay enqueues it again. Now the worker processes it twice. The customer gets two emails. How do you fix this?

const delivery = await pool.query(
  `SELECT status FROM notification_deliveries
   WHERE notification_id = $1 AND channel_id = $2`,
  [notification_id, channel.id]
);
if (delivery.rows[0]?.status === "sent") {
console.log("Already sent on this channel — skipping");
continue; // safe to skip this channel
}

The Outbox Status Machine

The Notification Delivery Status Machine

BullMQ — Retry With Backoff

One of the biggest v1 failures: if a notification errors, it's gone. BullMQ handles retry automatically:

import { Queue } from "bullmq";

// BullMQ queue — connected to Redis
const connection = { host: "localhost", port: 6379 };
export const notificationQueue = new Queue("notifications", { connection });

await notificationQueue.add(
  "send-notification",
  { notification_id, user_id },
  {
    jobId: `notif-${notification_id}`,  // deduplication
    attempts: 3,
    backoff: {
      type: "exponential",
      delay: 1000,  // 1s → 2s → 4s
    },
  }
);

No retry logic in your code. No manual re-enqueue. If the worker throws, BullMQ retries with exponential backoff. After 3 attempts, it goes to the dead-letter queue where you can inspect and replay.

V3 Sequence — The Same Order, Done Right

Load Test — V3

10 concurrent users, same degraded email (5s):

user #1  →  Fast 200 OK ✅
...
user #10 →  Fast 200 OK ✅

Workers processed all notifications in background.
The HTTP response is no longer coupled to notification-provider latency.

[Visual: Same bar chart as before, but now response times are flat at <50ms regardless of email speed or concurrent users. The email processing line is separate — in the background — and can spike without affecting the response time line]

Part 4 — How Real Apps Do It

Where Our V3 Fits in a Production Architecture

Our entire v3 notification system is one microservice in a production food delivery architecture. Here's how it fits:

Our v3 is the green box. The Order Service publishes to Kafka instead of writing to an outbox directly — Kafka is the durable log. The notification service consumes from Kafka and uses its own internal queue (like BullMQ) for per-channel dispatch.

What's Different at Scale

⚠️ Transparency note: The specifics below represent common architectural patterns at high-scale food delivery companies, consistent with publicly available engineering analyses. They are not direct quotes from official engineering posts — treat them as "patterns at this scale" rather than confirmed implementations.

One Real Named System — Swiggy's Klaxon

Swiggy has publicly documented Klaxon, an in-house real-time alerting platform, on their engineering blog Swiggy Bytes:

Klaxon processes over 1.5 million internal alerts daily using Complex Event Processing (CEP) — triggering automated SOPs when anomalies are detected (e.g., delayed orders, partner downtime).

Important distinction: Klaxon handles internal operational alerts (engineering and ops teams) — not customer-facing push/email notifications. It's a different audience for the same underlying problem. Worth knowing about because it shows that even internal notification systems need dedicated infrastructure at scale.

🤔 Stop and think: In our model, when order.placed fires, the notification manager explicitly knows to notify customer + restaurant. In a Kafka-based model, who decides that?

Summary — The Mental Models

V1 → V3 in One Diagram

Naive (v1)

Scaled (v3)

The Three Principles

1. Notifications are reactions, not actions. They should not block the user-facing request, but the business event that triggers them must be durably recorded. A notification system isn't primarily a sending problem; it's a reliability problem around asynchronous side effects.

2. Outbox + Idempotency = Reliable At-Least-Once Processing Outbox provides durable intent (at-least-once publication). Idempotency makes retries safe. True exactly-once delivery to external providers (like an email API) cannot be perfectly guaranteed, but this combination ensures safe, reliable processing without observable duplicates.

3. Each failure mode demands a separate mechanism

• Lost notification intent → Transactional Outbox

• Duplicate processing → Channel-level Idempotency keys

• Slow channel → Async workers

• Channel down → Retry with backoff

• Total failure → Dead-letter queue

Open Questions (For Your Next Deep Dive)

• [ ] Notification preferences: How does the user opt out of email but keep push? Where does that filter live — in the outbox, the worker, or the channel table?

• [ ] Fan-out at scale: If 1 million users follow a restaurant and it goes live — how do you fan-out 1M notifications without blocking? (Hint: fan-out on write vs fan-out on read)

• [ ] Ordering guarantees: Does it matter if a user gets the "order shipped" notification before "order placed"? How do you prevent out-of-order delivery?

• [ ] Read receipts: How does an app know you've seen a notification? What infrastructure enables that feedback loop?

• [ ] Multi-region: Your notification worker is in Mumbai. The user is in Delhi. The FCM token is registered to a US data center. How many network hops is that?

References

1. Transactional Outbox Pattern — Chris Richardson, Microservices.io https://microservices.io/patterns/data/transactional-outbox.html Canonical definition of the pattern, dual-write problem, relay strategies, and at-least-once delivery guarantees.

2. BullMQ Documentation — Rate Limiting, Job Priority, Concurrency [https://docs.bullmq.io\] Official docs for the queue library used in v3. See rate limiting and priority sections.

3. Microservices Patterns — Chris Richardson (Book, 2018) Manning Publications. ISBN: 978-1617294549 Chapter on messaging covers outbox, idempotency, and exactly-once processing in depth.

4. Enabling Real-Time Business Monitoring with Klaxon — Swiggy Bytes (Official Engineering Blog) [https://bytes.swiggy.com\] Swiggy's internal CEP-based alerting platform processing 1.5M alerts/day. Note: internal alerting, not customer-facing notifications.

5. Designing Scalable Notification Systems — Aditya Goel, Medium [https://adityagoel123.medium.com/designing-scalable-notification-system-79f83272755e\] Complementary system design overview covering requirements, SLAs, and architectural patterns.