The Outbox Pattern for Reliable Messaging

In distributed systems, you frequently need to update a database and publish a message to a broker atomically. Either both happen or neither happens. Without a distributed transaction (which you should avoid), the transactional outbox pattern provides the guarantee you need.

The Dual-Write Problem

Consider this common scenario: you save an order to the database and then publish an event to a message broker. If the publish fails after the save succeeds, your database has an order but downstream services never learn about it. If you reverse the order and publish first, the save might fail, and downstream services react to an order that does not exist.

// BROKEN — dual write problem
public async Task CreateOrderAsync(CreateOrderCommand command, CancellationToken ct)
{
    var order = new Order(command.CustomerId);
    _db.Orders.Add(order);
    await _db.SaveChangesAsync(ct); // Step 1: succeeds

    // Step 2: what if this fails? Database has the order, broker does not.
    await _messageBus.PublishAsync(new OrderCreatedEvent(order.Id), ct);
}

The Outbox Solution

Instead of publishing directly to the broker, write the message to an outbox table in the same database transaction as the business data. A separate background process reads the outbox and publishes to the broker.

public class OutboxMessage
{
    public long Id { get; set; }
    public Guid MessageId { get; set; } = Guid.NewGuid();
    public string Type { get; set; } = "";
    public string Payload { get; set; } = "";
    public DateTimeOffset CreatedAt { get; set; } = DateTimeOffset.UtcNow;
    public DateTimeOffset? ProcessedAt { get; set; }
}

public async Task CreateOrderAsync(CreateOrderCommand command, CancellationToken ct)
{
    var order = new Order(command.CustomerId);
    _db.Orders.Add(order);

    // Write to outbox in the SAME transaction
    _db.OutboxMessages.Add(new OutboxMessage
    {
        Type = nameof(OrderCreatedEvent),
        Payload = JsonSerializer.Serialize(new OrderCreatedEvent(order.PublicId))
    });

    await _db.SaveChangesAsync(ct); // Atomic — both or neither
}

The Outbox Publisher

A background service polls the outbox table and publishes pending messages to the broker. After successful publishing, it marks the message as processed.

public class OutboxPublisher(
    IServiceScopeFactory scopeFactory,
    IMessageBus messageBus,
    ILogger<OutboxPublisher> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            try
            {
                using var scope = scopeFactory.CreateScope();
                var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();

                var messages = await db.OutboxMessages
                    .Where(m => m.ProcessedAt == null)
                    .OrderBy(m => m.CreatedAt)
                    .Take(100)
                    .ToListAsync(ct);

                foreach (var message in messages)
                {
                    await messageBus.PublishRawAsync(message.Type, message.Payload, ct);
                    message.ProcessedAt = DateTimeOffset.UtcNow;
                }

                await db.SaveChangesAsync(ct);
            }
            catch (Exception ex)
            {
                logger.LogError(ex, "Outbox publishing failed, will retry");
            }

            await Task.Delay(TimeSpan.FromSeconds(5), ct);
        }
    }
}

Idempotent Consumers

Because the outbox publisher uses at-least-once delivery (a crash after publishing but before marking as processed will cause a re-publish), consumers must be idempotent. Use the MessageId for deduplication.

public class OrderCreatedHandler(AppDbContext db) : IMessageHandler<OrderCreatedEvent>
{
    public async Task HandleAsync(OrderCreatedEvent @event, MessageContext context, CancellationToken ct)
    {
        // Deduplication check
        if (await db.ProcessedMessages.AnyAsync(m => m.MessageId == context.MessageId, ct))
            return;

        // Process the event
        await ProcessOrderCreatedAsync(@event, ct);

        // Record that we processed this message
        db.ProcessedMessages.Add(new ProcessedMessage { MessageId = context.MessageId });
        await db.SaveChangesAsync(ct);
    }
}

Cleanup and Retention

The outbox table grows continuously. Implement a cleanup job that removes processed messages after a retention period. We keep processed messages for 7 days for debugging purposes, then delete them in batches to avoid table bloat.

The outbox pattern adds complexity but eliminates an entire class of data consistency bugs. In any system where database changes must reliably trigger external side effects, it is the correct solution.