Skip to content
ameoworks

Architecture · 2 min read

Send the message only after the save: reliable events between services

Why a message sent inside a database transaction can tell a lie, and the simple pattern that stops a rollback leaving a ghost message behind.

The problem

A service saves a new profile and then tells the rest of the system about it. The order of those two steps matters more than it looks.

If the service sends the message first and the save then fails, other services act on something that never happened. If it saves first and crashes before sending, nobody hears about a profile that does exist. Both are quiet bugs. Neither throws an error where you will see it.

What we do

Inside the transaction, the service only announces the change inside its own process, as a small internal event that carries ids and simple values. It does not touch the message broker at all.

A listener is tied to the after-commit moment of that transaction. Only once the database has really committed does the listener call the publisher that talks to RabbitMQ. If the transaction rolls back, the listener never runs, so no message leaves.

That gives us a few plain rules:

  • A transactional service method never calls the broker directly.
  • One listener class per aggregate, with one method per event. Spring finds the right method by the event type, so this is as safe as one class per event and much less code.
  • A listener does one job: call one publisher. It does not publish another internal event, because chains of after-commit events are hard to reason about.

Following one action across services

Reliable delivery is only half the story. When something goes wrong you want to follow one user action through every service it touched. For that, each request carries a correlation id: the entry filter accepts one or creates one, puts it in the logging context, and echoes it back on the response.

The lesson we learned the hard way is that this does not carry itself forward. Every kind of outbound call needs its own small hook that copies the id onto the next request or message: one for each HTTP client type, one when publishing to RabbitMQ, and one when a listener receives a message and restores the id. An internal audit found the id breaking at every hop after the first, even though each service had a correct filter. So we treat a new outbound client or listener without its hook as a defect, and we test the header and the restore.

The trade-off, honestly

After-commit publishing leaves one small gap. If the database commits and the process dies before the message is sent, that message is lost. For many events that is acceptable, because the next change or a periodic sync repairs the picture.

For events that must never be lost, the stronger answer is a transactional outbox: write the message into the same transaction as the data and have a separate process send it. It costs more moving parts, so it is worth adding only where the cost of a lost message is real.

What to take away

  • Never publish to a broker from inside a database transaction.
  • Publish after the commit, from a listener that does one thing.
  • Give every request a correlation id, and wire it into every kind of outbound call.
  • Know the gap you are accepting, and close it only where it matters.

Related services

More insights

Facing a similar problem?

Tell us what you are building or trying to fix. We will reply with an honest view.

Discuss your project