How to prevent duplicate SQS messages
Problem
In our system, queue processors must implement idempotency to prevent the double-processing of messages. Duplicate messages may arise in the following scenarios:
Scheduler and Message Producer: The scheduler or message producer may be triggered multiple times, occasionally rerunning due to timeouts.
Queue Management: If a lambda instance times out while processing a message, another instance may retrieve the same message if the visibility timeout is not properly set.
This can have terrible consequences. We aim to avoid sending duplicate emails or messages to our customers, not to mention inadvertently delivering duplicate gift cards.
So a generic idempotency mechanism is required.
