Consistent message delivery with Transactional Outbox Pattern
After a long run with this fun series of posts walking through a pet eCommerce project, I wanted to keep that as concise as possible. Still, there are subjects worth dedicating posts in more detail. So here I’ll talk about a pattern called Transactional Outbox that helps solve a known challenge when using events for exchanging data between microservices.
Sh** happens!
Everything works nicely until we have a problem that could prevent us from ensuring the operation’s atomicity. Writing to the database and publishing to a broker are two separate operations, and a crash between them breaks consistency in either direction:
- The database transaction can commit while the publish never happens (the event is lost).
- The publish can succeed while the database transaction later rolls back (a phantom event is published for a state that never existed).
This inconsistency is known as the dual-write problem, and it’s fairly simple to overcome it and ensure the message will be delivered at least once by implementing a pattern known as Outbox.
With a small step, you can go further.
Simply saying, in a scenario where we’re basing on events within a given transaction, and these events must be moved along for further processing, the pattern consists of persisting them into an outbox table rather than directly publishing the message to the message broker. It’s also essential that everything happens under the same database transaction that changes the aggregate, which causes the event we want to fan out. That way, everything happens simultaneously. If one of the counterparts fails, nothing should be partially committed. It’s all or nothing.
In another process, a background worker will fetch a batch of unprocessed messages and only then publish them to the message broker. Anything else remains the same with the Kafka consumer worker, and so on.
The OutboxMessages table
I built a simple structure with the following fields to demonstrate a basic structure for the outbox table:
1
2
3
4
5
6
7
Column | Type | Modifiers
--------------+--------------------------+-----------
Id | uuid | not null
Payload | text | not null
Type | text | not null
OcurredAt | timestamp with time zone | not null
ProcessedAt | timestamp with time zone | null
Producing messages
Alright, let’s play around with this simple implementation of a bug management system. The goal is to generate a Ticket to solve a bug, which can contain a title, description, and priority. Go to src/Outbox_101.Domain/Tickets, and you will find the Ticket aggregate root, which has a few actions allowing it to change its status. Every time the ticket is created, the initial status is OPEN, and it causes a TicketOpen domain event.
The same idea goes for SetInProgress() and Close(), which move the ticket to IN_PROGRESS and CLOSED states.
Also, there are two ConsoleApps that will start running simultaneously; Outbox_101.EventProducer and Outbox_101.EventConsumer. You can run everything in docker using:
1
docker compose up
Then, the producer calls a static procedure with the following code:
The code above does the following steps:
- A Ticket is built and causes a domain event.
- The ticket is staged through the
UnitOfWork.Ticketsrepository, which only tracks it in memory at this point. SaveChangesAsyncis called to commit the whole transaction, which is when the ticket is actually written to the table.
The code above does nothing more than get every uncommitted domain event from the currently tracked aggregates (in this case, Ticket). The OutboxUncommitedEvents() extension method builds an OutboxMessage object for each. Ultimately, the _dbContext.SaveChangesAsync() commits the whole transaction, which includes Tickets and their respective outboxed domain events. Having this ultimate logic in the SaveChangesAsync is not part of the pattern, but it certainly helps avoid rewriting this operation every time.
Processing outbox messages
We can use a background service to process the messages saved in the database. Look at src/Outbox_101.Infrastructure.Workers/Outbox/Polling/OutboxMessageProcessingWorker.cs. With the help of the OutboxMessageProcessor service (in src/Outbox_101.Infrastructure/Outbox/Polling/), it can fetch a batch of unprocessed messages in a polling loop, publish the message to Kafka, and finally set the message as processed so that it won’t be processed any longer. You must ensure that, for obvious reasons of not propagating it again.
Note the loop still needs its wait interval inside the loop body, otherwise it hammers the database with no pause between cycles.
💡 This demo never deletes or archives processed rows, so the table grows indefinitely. A real deployment would need a retention/cleanup strategy.
Consuming messages
The final component is the EventConsumer application, which also hosts a background service implementing a Kafka consumer, configured to read the subscribed Tickets topic, consume its messages, deserialize the payload, and publish them using the IEventDispatcher service. This dispatcher was implemented using MediatR, which provides the INotification interface where I based the EventBase class inheriting from, so when the notification is published, it will be handled by the TicketHandler subscribed to it.
💡 Notice this creates a cascade:
- Consuming
TicketOpencallsSetInProgress(), which raises a newTicketInProgressevent into the same outbox, which gets published and consumed in turn, callingClose()and raising aTicketClosedevent. - Both apps share the same database, so this is what produces the multiple messages you’ll see in the screenshots below, not three independent tickets.
The result of the whole flow is what you can see below:
At this point, we achieved the goal. Messages were delivered at least once, and you can experiment with shutting the Kafka container down before sending some more new messages to turn it on later. With this logic, once the broker is up and running again, it can catch up with unprocessed messages from the outbox table to be consumed further.
⚠️ “At least once” cuts both ways, though: the same crash window that guarantees a message won’t be lost can also cause it to be published twice, so the consumer side must be idempotent. In Ticket.cs, the SetInProgress() and Close() methods guard against this by only transitioning when the ticket is in the expected prior status, so redelivering a TicketOpen won’t fire a second TicketInProgress.
Change Data Capture (CDC) with Debezium
The first approach used database polling to read unprocessed messages from the OutboxMessages table, and that creates unnecessary overhead. What if we could do the same thing without constantly polling the database? What if, once the event is inserted into the OutboxMessages table, the message can be published to Kafka immediately? Meet Debezium, a log-based Change Data Capture (CDC) platform:
The Kafka Connect PostgreSQL Change Data Capture (CDC) Source connector (Debezium) for Confluent Cloud can obtain a snapshot of the existing data in a PostgreSQL database and then monitor and record all subsequent row-level changes to that data. The connector supports Avro, JSON Schema, Protobuf, or JSON (schemaless) output data formats. All of the events for each table are recorded in a separate Apache Kafka® topic. The events can then be easily consumed by applications and services.
That’s Confluent Cloud’s description of their managed connector. What this project runs is the self-hosted, open-source Debezium connector instead, but the core idea is the same: Debezium will free you from writing all that boilerplate for fetching, processing, and publishing messages. You can read and learn more from their documentation for more details.
Like everything in this project, the environment for this is defined in the docker-compose.yml file.
💡 Note: I had to configure Postgres to use wal_level=logical, as explained here.
Connector configuration
If you check Docker, you will notice the connect container running on PORT 8083, but it’s not configured yet. We need to configure the Outbox Event Router with Debezium. Read more about each configuration parameter in their documentation. You need just to PUT a connector config to http://localhost:8083/connectors/{connector-name}/config
If you’re already running the project, let’s start over with a few changes. You can skip this step otherwise.
- Stop and remove the containers with the following command:
1
docker compose down -v
I already put the JSON body with the configuration I used in the Outbox_101.EventProducer/debezium-ticket-connector.json, so the only thing you need to do is go to Outbox_101.EventProducer/Program.cs host configuration, there’s the AddOutboxSetup extension method with an optional parameter to start another background service that will PUT the configuration for Debezium instead of the workers we have used so far. All you need to do is set it to true and rerun the containers.
1
2
3
4
services
.AddHttpClient()
.AddPersistence(configuration)
.AddOutboxSetup(configuration, true);
Everything should work like before, except if checking the OutboxMessages table, the ProcessedAt field will now stay null forever: Debezium reads changes from the write-ahead log, not from the table itself, so it never marks a row as processed. It also never deletes rows, so just like the polling worker, the table keeps growing. In a real-life scenario, the canonical approach is to insert and delete the outbox row in the same transaction: Debezium’s connector captures the insert from the WAL, and the delete never reaches a consumer. This project is just a demo for the outbox pattern, so I used the same model interchangeably, although it’s different from what you’ll do in a real-life scenario.
I’m not using a schema registry here, but you can check this documentation to learn more about Avro serialization.
Final thoughts
This post showed a basic implementation of the Outbox pattern to guarantee message delivery between applications, using events to exchange data integrally and demonstrating publishing and consumption through two different techniques.
Bear in mind that you can use any of the many message brokers available to stream messages, since the pattern isn’t technology-specific. Deciding whether to use DB polling or CDC is also a trade-off worth weighing, since CDC can have more limited support and require extra configuration in the database and message streamer.