The Outbox Pattern, Step by Step

The Problem#

Many services need to update a database and publish an event. If the database write succeeds but the publish fails, downstream systems never hear about the change.

Why It Matters#

The outbox pattern avoids a distributed transaction between your database and your message broker. Instead, it uses the database transaction you already trust, then publishes events asynchronously.

This pattern appears in real event-driven architectures across ordering, billing, fulfillment, notification, and audit systems. It is especially useful when using Kafka, RabbitMQ, SNS/SQS, or any broker that cannot participate in the same transaction as your database.

Project Example#

An order service inserts an order and must publish order.created for fulfillment. Instead of publishing directly inside the request, write the order and an outbox row in the same database transaction. A worker later reads the outbox and publishes the event.

CREATE TABLE outbox_events (
  id uuid PRIMARY KEY,
  topic text NOT NULL,
  payload jsonb NOT NULL,
  published_at timestamptz
);

Implementation Example#

BEGIN;

INSERT INTO orders (id, customer_id, total_cents)
VALUES ($1, $2, $3);

INSERT INTO outbox_events (id, topic, payload)
VALUES (
  gen_random_uuid(),
  'order.created',
  jsonb_build_object('orderId', $1, 'customerId', $2)
);

COMMIT;

The worker can poll unpublished rows with FOR UPDATE SKIP LOCKED, publish them, and mark them as published after broker acknowledgment.

Implementation Checklist#

• Insert business data and outbox event in one transaction.
• Poll unpublished events in small batches.
• Publish with idempotent event ids.
• Mark events as published only after broker acknowledgment.
• Add retry count and dead-letter handling.
• Keep event schema versioned.
• Monitor oldest unpublished event age.
• Make consumers idempotent because brokers can redeliver.

Production Notes#

Outbox tables need retention. Keep enough history for debugging and replay, but do not let years of published events remain in the hot table. Archive or delete old published rows on a schedule.

Common Mistakes#

• Publishing before the database commit.
• Deleting outbox rows too early for debugging.
• Not making consumers idempotent.
• Letting the outbox table grow without retention.
• Treating "published" as successful before the broker confirms it.

Summary#

The outbox pattern gives reliable event publishing without distributed transactions. It is a practical default for event-driven services.