Reliable event delivery for document workflows
My contribution. I implemented a transactional outbox in Python, a relay that retries publication to RabbitMQ, and idempotent consumers that prevent duplicate deliveries from applying the same workflow update twice.
Failure test results. I reproduced both scenarios in a standalone demo using Docker Compose and Kubernetes.
- RabbitMQ was offline during a write. The API returned HTTP 201, the document was saved in PostgreSQL, and the event stayed pending. After RabbitMQ restarted, the event was delivered and one history entry appeared.
- The consumer stopped after commit, before acknowledgement. RabbitMQ redelivered the event to a replacement consumer. The queue drained and the document still had exactly one history entry.
The problem
A document update had to save the change in PostgreSQL and notify other services through RabbitMQ. If the database commit succeeded and the process stopped before sending the message, the document would show its new state while downstream services would have no record of the change.
How the outbox works
The service writes the document change and its outgoing event in the same PostgreSQL transaction. After commit, a separate relay publishes the event to RabbitMQ. If publication fails, the event stays pending for another attempt.
- First successful processing → apply the workflow update.
- Already processed → finish without repeating the update.
Handling redelivery and duplicates
Duplicates can come from either side. The relay can publish an event again if it stops after RabbitMQ accepts the message but before it records successful publication. RabbitMQ can also redeliver a message if the consumer disconnects before acknowledging it.
The consumer records the event ID and writes the history entry in the same PostgreSQL transaction, then acknowledges the message. In the demo, I stopped the first consumer between commit and acknowledgement. The replacement consumer found the existing event ID, logged duplicate_skipped, and acknowledged the message without writing a second history entry.