Aleksei Kokorev

Backend engineering · Event-driven systems

Reliable event delivery for document workflows

My contribution. I implemented a transactional outbox in Python, a relay that retries publication to RabbitMQ, and idempotent consumers that prevent duplicate deliveries from applying the same workflow update twice.

Failure test results. I reproduced both scenarios in a standalone demo using Docker Compose and Kubernetes.

The problem

A document update had to save the change in PostgreSQL and notify other services through RabbitMQ. If the database commit succeeded and the process stopped before sending the message, the document would show its new state while downstream services would have no record of the change.

How the outbox works

The service writes the document change and its outgoing event in the same PostgreSQL transaction. After commit, a separate relay publishes the event to RabbitMQ. If publication fails, the event stays pending for another attempt.

From document update to downstream processing
Python document service Receive a document update.
↓ Write both records in one transaction
PostgreSQL Commit the document change and its outgoing event together.
↓ Read committed outbox events
Outbox processor Publish each event. If publication fails, keep it pending and retry later.
↓ Publish
RabbitMQ Queue the message for delivery. If a consumer disconnects before acknowledging it, deliver it again.
↓ Delivery or redelivery
Idempotent consumer
  • First successful processing → apply the workflow update.
  • Already processed → finish without repeating the update.

Handling redelivery and duplicates

Duplicates can come from either side. The relay can publish an event again if it stops after RabbitMQ accepts the message but before it records successful publication. RabbitMQ can also redeliver a message if the consumer disconnects before acknowledging it.

The consumer records the event ID and writes the history entry in the same PostgreSQL transaction, then acknowledges the message. In the demo, I stopped the first consumer between commit and acknowledgement. The replacement consumer found the existing event ID, logged duplicate_skipped, and acknowledged the message without writing a second history entry.

Run the demo