AWS SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service for decoupling and scaling microservices, distributed systems, and serverless applications.

Core Concepts
Common Patterns
CLI Reference
Best Practices
Troubleshooting
References

Core Concepts

Queue Types

Type	Description	Use Case
Standard	At-least-once, best-effort ordering	High throughput
FIFO	Exactly-once, strict ordering	Order-sensitive processing

Key Settings

Setting	Description	Default
Visibility Timeout	Time message is hidden after receive	30 seconds
Message Retention	How long messages are kept	4 days (max 14)
Delay Seconds	Delay before message is available	0
Max Message Size	Maximum message size	256 KB

Dead-Letter Queue (DLQ)

Queue for messages that failed processing after maxReceiveCount attempts.

Common Patterns

Create a Standard Queue

AWS CLI:

aws sqs create-queue \
  --queue-name my-queue \
  --attributes '{
    "VisibilityTimeout": "60",
    "MessageRetentionPeriod": "604800",
    "ReceiveMessageWaitTimeSeconds": "20"
  }'

boto3:

import boto3

sqs = boto3.client('sqs')

response = sqs.create_queue(
    QueueName='my-queue',
    Attributes={
        'VisibilityTimeout': '60',
        'MessageRetentionPeriod': '604800',
        'ReceiveMessageWaitTimeSeconds': '20'  # Long polling
    }
)
queue_url = response['QueueUrl']

Create FIFO Queue

aws sqs create-queue \
  --queue-name my-queue.fifo \
  --attributes '{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "true"
  }'

Configure Dead-Letter Queue

# Create DLQ
aws sqs create-queue --queue-name my-queue-dlq

# Get DLQ ARN
DLQ_ARN=$(aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue-dlq \
  --attribute-names QueueArn \
  --query 'Attributes.QueueArn' --output text)

# Set redrive policy on main queue
aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --attributes "{
    \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${DLQ_ARN}\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\"
  }"

Send Messages

import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'

# Send single message
sqs.send_message(
    QueueUrl=queue_url,
    MessageBody=json.dumps({'order_id': '12345', 'action': 'process'}),
    MessageAttributes={
        'MessageType': {
            'DataType': 'String',
            'StringValue': 'Order'
        }
    }
)

# Send to FIFO queue
sqs.send_message(
    QueueUrl='https://sqs.us-east-1.amazonaws.com/123456789012/my-queue.fifo',
    MessageBody=json.dumps({'order_id': '12345'}),
    MessageGroupId='order-12345',
    MessageDeduplicationId='unique-id-12345'
)

# Batch send (up to 10 messages)
sqs.send_message_batch(
    QueueUrl=queue_url,
    Entries=[
        {'Id': '1', 'MessageBody': json.dumps({'id': 1})},
        {'Id': '2', 'MessageBody': json.dumps({'id': 2})},
        {'Id': '3', 'MessageBody': json.dumps({'id': 3})}
    ]
)

Receive and Process Messages

import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'

while True:
    # Long polling (wait up to 20 seconds)
    response = sqs.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=10,
        WaitTimeSeconds=20,
        MessageAttributeNames=['All'],
        AttributeNames=['All']
    )

    messages = response.get('Messages', [])

    for message in messages:
        try:
            body = json.loads(message['Body'])
            print(f"Processing: {body}")

            # Process message...

            # Delete on success
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=message['ReceiptHandle']
            )
        except Exception as e:
            print(f"Error processing message: {e}")
            # Message will become visible again after visibility timeout

Lambda Integration

# Create event source mapping
aws lambda create-event-source-mapping \
  --function-name my-function \
  --event-source-arn arn:aws:sqs:us-east-1:123456789012:my-queue \
  --batch-size 10 \
  --maximum-batching-window-in-seconds 5

Lambda handler:

def handler(event, context):
    for record in event['Records']:
        body = json.loads(record['body'])
        message_id = record['messageId']

        try:
            process_message(body)
        except Exception as e:
            # Raise to put message back in queue
            raise

    return {'batchItemFailures': []}

CLI Reference

Queue Management

Command	Description
`aws sqs create-queue`	Create queue
`aws sqs delete-queue`	Delete queue
`aws sqs list-queues`	List queues
`aws sqs get-queue-url`	Get queue URL by name
`aws sqs get-queue-attributes`	Get queue settings
`aws sqs set-queue-attributes`	Update queue settings

Messaging

Command	Description
`aws sqs send-message`	Send single message
`aws sqs send-message-batch`	Send up to 10 messages
`aws sqs receive-message`	Receive messages
`aws sqs delete-message`	Delete message
`aws sqs delete-message-batch`	Delete up to 10 messages
`aws sqs purge-queue`	Delete all messages

Visibility

Command	Description
`aws sqs change-message-visibility`	Change timeout
`aws sqs change-message-visibility-batch`	Batch change

Best Practices

Message Processing

Use long polling (WaitTimeSeconds=20) to reduce API calls
Delete messages promptly after successful processing
Configure appropriate visibility timeout (> processing time)
Implement idempotent consumers for at-least-once delivery

Dead-Letter Queues

Always configure DLQ for production queues
Set appropriate maxReceiveCount (usually 3-5)
Monitor DLQ depth with CloudWatch alarms
Process DLQ messages manually or with automation

FIFO Queues

Use message group IDs to partition ordering
Enable content-based deduplication or provide dedup IDs
Throughput: 300 msgs/sec without batching, 3000 with

Security

Use queue policies to control access
Enable encryption with SSE-SQS or SSE-KMS
Use VPC endpoints for private access

Troubleshooting

Messages Not Being Received

Causes:

Short polling returning empty
All messages in flight (visibility timeout)
Messages delayed (DelaySeconds)

Debug:

# Check queue attributes
aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names All

# Check approximate message counts
aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names \
    ApproximateNumberOfMessages,\
    ApproximateNumberOfMessagesNotVisible,\
    ApproximateNumberOfMessagesDelayed

Messages Going to DLQ