Amazon Simple Queue Service (Amazon SQS) is a reliable, highly-scalable hosted queue for storing messages as they travel between applications or microservices. Amazon SQS moves data between distributed application components and helps you decouple these components.
There are two types of queues:
1. Standard queue
2. FIFO queue
SQS offers two types of queues – Standard & FIFO queues
Message Order
- Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they are sent. occasionally (because of the highly-distributed architecture that allows high throughput), more than one copy of a message might be delivered out of order
- FIFO queues offer first-in-first-out delivery and exactly-once processing: the order in which messages are sent and received is strictly preserved
Delivery
- Standard queues guarantee that a message is delivered at least once and duplicates can be introduced into the queue
- FIFO queues ensure a message is delivered exactly once and remains available until a consumer processes and deletes it; duplicates are notintroduced into the queue
Transactions Per Second (TPS)
- Standard queues allow nearly-unlimited number of transactions per second
- FIFO queues are limited to 300 transactions per second per API action
Regions
- Standard queues are available in all the regions
- FIFO queues are currently available in limited regions including the US West (Oregon), US East (Ohio), US East (N. Virginia), and EU (Ireland)
SQS Buffered Asynchronous Client
- FIFO queues aren’t currently compatible with the SQS Buffered Asynchronous Client, where messages are buffered at client side and send as a single request to the SQS queue to reduce cost.
AWS Services Supported
- Standard Queues are supported by all AWS services
- FIFO Queues are currently not supported by all AWS services like
- CloudWatch Events
- S3 Event Notifications
- SNS Topic Subscriptions
- Auto Scaling Lifecycle Hooks
- AWS IoT Rule Actions
- AWS Lambda Dead Letter Queues
Use Cases
- Standard queues can be used in any scenarios, as long as the application can process messages that arrive more than once and out of order
- Decouple live user requests from intensive background work: Let users upload media while resizing or encoding it.
- Allocate tasks to multiple worker nodes: Process a high number of credit card validation requests.
- Batch messages for future processing: Schedule multiple entries to be added to a database.
- FIFO queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated
- Ensure that user-entered commands are executed in the right order.
- Display the correct product price by sending price modifications in the right order.
- Prevent a student from enrolling in a course before registering for an account.
SQS APIs: two major categories, queue management and message management
FIFO queue specific API parameters:
MessageDeduplicationId
This parameter applies only to FIFO (first-in-first-out) queues.
The token used for deduplication of messages within a 5-minute minimum deduplication interval. If a message with a particular MessageDeduplicationId
is sent successfully, subsequent messages with the same MessageDeduplicationId
are accepted successfully but aren't delivered. For more information, see Exactly-Once Processing in the Amazon Simple Queue Service Developer Guide.
-
Every message must have a unique MessageDeduplicationId
,
-
You may provide a MessageDeduplicationId
explicitly.
-
If you aren't able to provide a MessageDeduplicationId
and you enable ContentBasedDeduplication
for your queue, Amazon SQS uses a SHA-256 hash to generate the MessageDeduplicationId
using the body of the message (but not the attributes of the message).
-
If you don't provide a MessageDeduplicationId
and the queue doesn't have ContentBasedDeduplication
set, the action fails with an error.
-
If the queue has ContentBasedDeduplication
set, your MessageDeduplicationId
overrides the generated one.
-
When ContentBasedDeduplication
is in effect, messages with identical content sent within the deduplication interval are treated as duplicates and only one copy of the message is delivered.
-
If you send one message with ContentBasedDeduplication
enabled and then another message with a MessageDeduplicationId
that is the same as the one generated for the first MessageDeduplicationId
, the two messages are treated as duplicates and only one copy of the message is delivered.
Note
The MessageDeduplicationId
is available to the recipient of the message (this can be useful for troubleshooting delivery issues).
If a message is sent successfully but the acknowledgement is lost and the message is resent with the sameMessageDeduplicationId
after the deduplication interval, Amazon SQS can't detect duplicate messages.
Amazon SQS continues to keep track of the message deduplication ID even after the message is received and deleted.
The length of MessageDeduplicationId
is 128 characters. MessageDeduplicationId
can contain alphanumeric characters (a-z
, A-Z
, 0-9
) and punctuation (!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
).
MessageGroupId
This parameter applies only to FIFO (first-in-first-out) queues.
The tag that specifies that a message belongs to a specific message group. Messages that belong to the same message group are processed in a FIFO manner (however, messages in different message groups might be processed out of order). To interleave multiple ordered streams within a single queue, use MessageGroupId
values (for example, session data for multiple users). In this scenario, multiple readers can process the queue, but the session data of each user is processed in a FIFO fashion.
-
You must associate a non-empty MessageGroupId
with a message. If you don't provide a MessageGroupId
, the action fails.
-
ReceiveMessage
might return messages with multiple MessageGroupId
values. For each MessageGroupId
, the messages are sorted by time sent. The caller can't specify a MessageGroupId
.
The length of MessageGroupId
is 128 characters. Valid values are alphanumeric characters and punctuation (!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~)
.
Increasing Throughput with Horizontal Scaling and Batching
Amazon SQS queues can deliver very high throughput (many thousands of messages per second). To achieve this throughput, you must scale message producers and consumers horizontally (add more producers and consumers).
In addition to horizontal scaling, batching provides a throughput with fewer threads, connections, and requests than would be required by individual message requests. You can use batched Amazon SQS API actions to send, receive, or delete up to 10 messages at a time. Because Amazon SQS charges by the request instead of by the message, batching can also substantially reduce costs.
Limits Related to Queues
The following table lists limits related to queues.
Limit | Description |
---|
Inflight messages per queue | For standard queues, there can be a maximum of 120,000 inflight messages per queue. If you reach this limit, Amazon SQS returns the OverLimit error message. To avoid reaching the limit, you should delete messages from the queue after they're processed. You can also increase the number of queues you use to process your messages. |
For FIFO queues, there can be a maximum of 20,000 inflight messages per queue. If you reach this limit, Amazon SQS returns no error messages. |
Queue name | A queue name can have up to 80 characters. The following characters are accepted: alphanumeric characters, hyphens (- ), and underscores (_ ).
Note
Queue names are case-sensitive (for example, Test-queue and test-queue are different queues).
|
The name of a FIFO queue must end with the .fifo suffix. The suffix counts towards the 80-character queue name limit. To determine whether a queue is FIFO, you can check whether the queue name ends with the suffix. |
Queue tag | We don't recommend adding more than 50 tags to a queue. |
The tag Key is required, but the tag Value is optional. |
The tag Key and tag Value are case-sensitive. |
The tag Key and tag Value can include Unicode alphanumeric characters in UTF-8 and whitespaces. The following special characters are allowed: _ . : / = + - @ |
The tag Key or Value must not include the reserved prefix aws: (you can't delete tag keys or values with this prefix). |
The maximum tag Key length is 128 Unicode characters in UTF-8. The tag Key must not be empty or null. |
The maximum tag Value length is 256 Unicode characters in UTF-8. The tag Value may be empty or null. |
Tagging API actions are limited to 5 TPS per AWS account. If your application requires a higher throughput, file a technical support request. |
Limits Related to Messages
The following table lists limits related to messages.
Limit | Description |
---|
Message attributes | A message can contain up to 10 metadata attributes. |
Message batch | A single message batch request can include a maximum of 10 messages. For more information, see Advanced Configuration in the Amazon SQS Batch API Actions section. |
Message content | A message can include only XML, JSON, and unformatted text. The following Unicode characters are allowed: #x9 | #xA | #xD | #x20 to #xD7FF | #xE000 to #xFFFD | #x10000 to #x10FFFF Any characters not included in this list are rejected. For more information, see the W3C specification for characters. |
Message retention | By default, a message is retained for 4 days. The minimum is 60 seconds (1 minute). The maximum is 1,209,600 seconds (14 days). |
Message throughput | Standard queues support a nearly unlimited number of transactions per second (TPS) per API action. |
-
FIFO queues support up to 300 messages per second (300 send, receive, or delete operations per second). -
When you batch 10 messages per operation (maximum), FIFO queues can support up to 3,000 messages per second. To request a limit increase, file a support request.
|
Message size | The minimum message size is 1 byte (1 character). The maximum is 262,144 bytes (256 KB). To send messages larger than 256 KB, you can use the Amazon SQS Extended Client Library for Java. This library allows you to send an Amazon SQS message that contains a reference to a message payload in Amazon S3. The maximum payload size is 2 GB. |
Message visibility timeout | The default visibility timeout for a message is 30 seconds. The maximum is 12 hours. |
Policy information | The maximum limit is 8,192 bytes, 20 statements, 50 principals, or 10 conditions. For more information, see Limits Related to Policies. |
Using Amazon SQS Dead-Letter Queues
Amazon SQS supports dead-letter queues. A dead-letter queue is a queue that other (source) queues can target for messages that can't be processed (consumed) successfully. Dead-letter queues are useful for debugging your application or messaging system. Dead-letter queues allow you to isolate problematic messages to determine why their processing doesn't succeed.
essages Using Short Polling
The behavior of consuming messages from the queue depends on whether you use short (standard) polling, the default behavior, or long polling. For more information about long polling, see Amazon SQS Long Polling.
When you consume messages from the queue using short polling, Amazon SQS samples a subset of the servers (based on a weighted random distribution) and returns messages from just these servers. Thus, a particular receive request might not return all of your messages. However, if you have a small number of messages in your queue (fewer than 1,000), one particular request might not return any of your messages, whereas a subsequent request returns them. If you keep consuming from your queues, Amazon SQS samples all of the servers, and you receive all your messages.
The following figure shows the short-polling behavior of messages returned after one of your system components makes a receive request. Amazon SQS samples several of the servers (in gray) and returns the messages from those servers (Message A, C, D, and B). Message E isn't returned to this particular request, but is returned to a subsequent request.
Amazon SQS Long Polling
Long polling helps reduce your cost of using Amazon SQS by reducing the number of empty responses (when there are no messages available to return in reply to a ReceiveMessage
request sent to an Amazon SQS queue) and eliminating false empty responses (when messages are available in the queue but aren't included in the response):
-
Long polling reduces the number of empty responses by allowing Amazon SQS to wait until a message is available in the queue before sending a response. Unless the connection times out, the response to the ReceiveMessage
request contains at least one of the available messages, up to the maximum number of messages specified in the ReceiveMessage
action.
-
Long polling eliminates false empty responses by querying all (rather than a limited number) of the servers.
-
Long polling returns messages as soon any message becomes available.
Message ID
Each message receives a system-assigned message ID that Amazon SQS returns to you in the SendMessage
response. This identifier is useful for identifying messages. (However, to delete a message you need the message's receipt handle.) The maximum length of a message ID is 100 characters.
Receipt Handle
Every time you receive a message from a queue, you receive a receipt handle for that message. This handle is associated with the action of receiving the message, not with the message itself. To delete the message or to change the message visibility, you must provide the receipt handle (not the message ID). Thus, you must always receive a message before you can delete it (you can't put a message into the queue and then recall it). The maximum length of a receipt handle is 1024 characters.
Important
If you receive a message more than once, each time you receive it, you get a different receipt handle. You must provide the most recently received receipt handle when you request to delete the message (otherwise, the message might not be deleted).
Additional Identifiers for FIFO Queues
For more information about the following identifiers, see Exactly-Once Processing and the Amazon Simple Queue Service API Reference.
Message Deduplication ID
The token used for deduplication of sent messages. If a message with a particular message deduplication ID is sent successfully, any messages sent with the same message deduplication ID are accepted successfully but aren't delivered during the 5-minute deduplication interval.
Message Group ID
The tag that specifies that a message belongs to a specific message group. Messages that belong to the same message group are always processed one by one, in a strict order relative to the message group (however, messages that belong to different message groups might be processed out of order).
Sequence Number
The large, non-consecutive number that Amazon SQS assigns to each message.
Visibility Timeout
When a consumer receives and processes a message from a queue, the message remains in the queue. Amazon SQS doesn't automatically delete the message. Because Amazon SQS is a distributed system, there's no guarantee that the consumer actually receives the message (for example, due to a connectivity issue, or due to an issue in the consumer application). Thus, the consumer must delete the message from the queue after receiving and processing it.
Immediately after the message is received, it remains in the queue. To prevent other consumers from processing the message again, Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consumers from receiving and processing the message. The default visibility timeout for a message is 30 seconds. The maximum is 12 hours.