django-celery-sqs's Introduction

Celery-SQS

Sample Django application that shows how to use Celery with Amazon SQS as the Broker.

Quick guide

Python 3.7+ is required
Clone this repo
Create a virtualenv
Install requirements

python3 -m venv celery-sqs
source celery-sqs/bin/activate
pip install -r requirements.txt

Add celerysqs/secret.py with the following template

Note

The SQS keys must have both IAM Policies listed below

KEY = "some secret key"  # for Django

SQS = {
    "access_key": "your aws_access_key",
    "secret_key": "your aws_secret_key",
}

Django Commands

run the server

$> DJANGO_SETTINGS_MODULE=celerysqs.conf.aws python manage.py runserver

run the celery worker

$> DJANGO_SETTINGS_MODULE=celerysqs.conf.aws celery worker -A celerysqs --concurrency=1 -l info

queue some tasks
```
$> ./scripts/curls.sh
```

IAM Policies

Note

Both policy types are required

(1) ListQueues (*)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1234567890000",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

(2) CRUD (prefix)

Note

- Replace {region} with your AWS Region - Replace {prefix} desired queue prefix

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1234567890000",
            "Effect": "Allow",
            "Action": [
                "sqs:ChangeMessageVisibility",
                "sqs:CreateQueue",
                "sqs:DeleteMessage",
                "sqs:DeleteQueue",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:SetQueueAttributes"
            ],
            "Resource": [
                "arn:aws:sqs:{region}:*:{prefix}*"
            ]
        }
    ]
}

Recommend Projects