Events and Queues: Automation That Never Loses a Message
Why doing the work inside the request fails, and how to move it to a queue without losing visibility or ordering.
"On purchase: record the order, send an email, add the learner to a list, notify the customer's system." Writing that inline in the request handler works perfectly — until the email provider takes a few seconds and checkout feels slow, or the last call fails and the whole thing rolls back despite a successful payment.
Separate what must happen now
Inside the request: what the user cannot proceed without — recording the purchase and granting access. Outside it: anything that can happen a second later unnoticed — email, syncing, reporting. The rule is simple: if its failure should not fail the operation, it does not belong in the request.
A queue changes which guarantees you have
- At-least-once delivery is the common default: a message may arrive twice, so the handler must be safely repeatable.
- Ordering is usually not guaranteed; if you need it, enforce it with a partition key rather than hoping.
- Every message gets a bounded attempt count, and what exceeds it goes to a dead-letter queue rather than nowhere.
- Lag is a first-class metric: a backing-up queue is worse than a failing one, because it fails silently.
Retry with exponential backoff
Retrying immediately doubles the load on an already struggling service and finishes it off. Space attempts increasingly, and add jitter so every worker does not retry at the same instant. Distinguish transient failures worth retrying from permanent ones that are not — an invalid email address will not become valid on the fifth attempt.
A dead-letter queue is not an admission of defeat. It is the difference between an error you can see and an error that vanishes.
Visibility beats elegance
The worst property of asynchronous work is that it fails away from the user's view. Give every job an id traceable back to the originating event, record attempt count and last failure reason, and build a plain screen showing backlog and failures. Without it you learn about problems from a customer.
When you do not need a queue
Do not build an event architecture for three background jobs. Start with the simplest thing that moves work off the request, and graduate to multiple prioritized queues when slow work starts competing with time-sensitive work — not before.
The best thing written on exponential backoff and jitter is the AWS Builders' Library article on timeouts, retries and backoff, with measurements rather than general advice. For the same patterns under stable names, Enterprise Integration Patterns remains the reference, and the Google SRE book covers handling overload.
If you are building: the most common mistake here is assuming a message arrives exactly once. Talk to us.
FAQ
How do I make a handler safely repeatable?
Give every event a unique id, record what you processed, and check before executing. Or design the operation so running it twice produces the same result.
How many retries before dead-lettering?
Five attempts spread over hours covers transient outages; what fails past that needs intervention, not a sixth attempt.
Do I need a dedicated message broker?
Not at first. A jobs table in your own database with correct locking carries you a long way and is far simpler to operate.