The DLCS Composite Handler is an implementation of DLCS RFC011.

The component is written in Python and utilises Django with the following extensions:

• Django REST Framework
• Django Q
• django-environ
• django-health-check

Additionally, the project uses:

• Boto3
• jsonschema
• pdf2image
• requests
• tqdm

The project ships with a docker-compose.yml that can be used to get a local version of the component running:

docker compose up

Note that for the Composite Handler to be able to interact with the target S3 bucket, the Docker Compose assumes that the AWS_PROFILE environment variable has been set and a valid AWS session is available.

This will create a PostgreSQL instance, bootstrap it with the required tables, deploy a single instance of the API, and three instances of the engine. Requests can then be targetted at localhost:8000.

The component can also be run directly, either in an IDE or from the CLI. The component must first be configured either via the creation of a .env file (see .env.dist for an example configuration), or via a set of environment variables (see the Configuration section).

Once configuration is in place, the following commands will start the API and / or engine:

• API: python manage.py runserver 0.0.0.0:8000
• Engine: python manage.py qcluster

Should the required tables not exist in the target database, the following commands should be run first:

python manage.py migrate
python manage.py createcachetable

Once the API is running, an administrator interface can be accessed via the browser at http://localhost:8000/admin. To create an administrator login, run the following command:

python manage.py createsuperuser

The administrator user can be used to browse the database and manage the queue (including deleting tasks and resubmitting failed tasks into the queue).

There are 3 possible entrypoints to make the above easier:

• entrypoint.sh - this will wait for Postgres to be available and run manage.py migrate and manage.py createcachetable if MIGRATE=True. It will run manage.py createsuperuser is INIT_SUPERUSER=True (also needs DJANGO_SUPERUSER_* envvars)
• entrypoint-api.sh - this runs above then starts nginx instance fronting gunicorn process
• entrypoint-worker.sh - this runs above then python manage.py qcluster

The following list of environment variables are supported:

Note that in order to access the S3 bucket, the Composite Handler assumes that valid AWS credentials are available in the environment - this can be in the former of environment variables, or in the form of ambient credentials.

By default Django Q will use the default Django ORM broker.

The SQS broker can be configured by specifying the SQS_BROKER_QUEUE_NAME environment variable. Default SQS broker behaviour is to create this queue if it is not found.

As with S3, above, Composite Handler assumes that valid AWS credentials are available in the environment.

The project ships with a Dockerfile:

docker build -t dlcs/composite-handler:local .

This will produce a single image that can be used to execute any of the supported Django commands, including running the API and the engine:

docker run dlcs/composite-handler:local python manage.py migrate # Apply any pending DB schema changes
docker run dlcs/composite-handler:local python manage.py createcachetable # Create the cache table (if it doesn't exist)
docker run --env-file .env -it --rm dlcs/composite-handler:local /srv/dlcs/entrypoint-api.sh # Run the API
docker run --env-file .env -it --rm dlcs/composite-handler:local /srv/dlcs/entrypoint-worker.sh # Run the engine
docker run dlcs/composite-handler:local python manage.py qmonitor # Monitor the workers