Amazon SQS (Simple Queue Service)

sqs.md

Amazon SQS

Standard Queue

• Oldest offering (over 10 years old)

• Fully managed service, used to decouple applications

• Attributes

  • Unlimited throughput, unlimited number of messages in queue
  • Default retention of messages: 4 days, maximum of 14 days
  • Low latency (<10ms on publish and receive)
  • Limiteation of 256KB per message sent

• Can have duplicated messages (at least once delivery, occassionally)

• Can have out of order messages (best effort ordering)

Producing Messages

• Produced to SQS using the SDK (SendMessage API)

• The message is persisted in SQS until a consumer deletes it

• Message retention: default 4 days, upto 14 days

• Example: send an order to be processed

  • Order Id
  • Customer Id
  • Any attributes you want

• SQS standard: unlimited throughput

Message Visibility Timeout

• After a message is polled by a consumer, it becomes invisible to other consumers

• By default, the "message visibility timeout" is 30 seconds

• That means the message has 30 seconds to be processed

• After the message visibility timeout is over, the message is "visible" in SQS yet again

• If a message is not processed within the visibility timeout, it will be processed twice

• A consumer could call the ChangeMessageVisibility API to get more time

• If visibility timeout is high (hours), and consumer crashes, re-processing will take time

• If visibility timeout is too low (seconds), we may get duplicates

Dead Letter Queue

• If a consumer fails to process a message within the VisibilityTimeout period, the message goes back to the queue

• We can set a threshold of how many times a message can go back to the queue

• After the MaximumReceives threshold is exceeded, the message goes into a dead letter queue for debugging purposes

• Make sure to process the messages in the DLQ before they expire

  • Good to set a retention period of 14 days for DLQ

Delay Queue

• Delay a message (consumers don't see it immediately) up to 15 mins
• Default is 0 seconds (message is available right away)
• Can set a default at queue level
• Can override the default on send using the DelaySeconds parameter

Long Polling

• When a consumer requests messages from the queue, it can optionally "wait" for messages to arrive if there are none in the queue
• This is called Long Polling
• Long Polling decreases the number of API calls made to SQS while increasing the efficiency and latency of your application
• The wait time can be between 1 sec to 20 secs (20 secs preferrable)
• Long Polling is preferable to Short Polling
• Long Polling can be enabled at the queue level or at the API level using WaitTimeSeconds

Extended Client

• Message size limit is 256KB, how to send large messages, eg 1GB?
• Using the SQS Extended Client (Java library)
  • Send large message to S3
  • Send the small metadata associated to that large message to SQS

FIFO Queue

• First In First Out (ordering of messages in the queue)
• Limited throughout: 300 msg/sec without batching. 3000 msg/sec with
• Exactly-once send capability (by removing duplicates)
• Messages are processed in order by the consumer

SQS FIFO - Deduplication

• Deduplication: The elimination of duplicate or redundant information
• De-duplication interval is 5 mins
• 2 de-duplication methids:
  • Content-based deduplication: will do a SHA-256 hash of the message body and compare
  • Message Deduplication ID: Explicitly provide a message deduplication id

SQS FIFO - Message Grouping

• If you specify the same value of MessageGroupID in an SQS FIFO queue, you can only have one consumer, and all the messages are in order
• To get ordering at the level of a subset of messages, specify different values for MessageGroupID
  • Messages that share a common MessageGroupID will be in order within the group
  • Each Group ID can have a different consumer (parallel processing!)
  • Ordering across groups is not guaranteed