This open source Terraform module provisions the necessary services to provide a data product on AWS.

• AWS S3
• AWS Athena
• AWS Glue
• AWS Lambda

module my_data_product {
  source = "[email protected]:datamesh-architecture/terraform-dataproduct-aws-athena.git"

  domain   = "<data_product_domain>"
  name     = "<data_product_name>"
  
  schedule = "0 0 * * ? *" # Run at 00:00 am (UTC) every day

  input = [
    {
      source = "<existing_s3_bucket>"
    }
  ]

  transform = {
    query    = "sql/<name_of_the_transform>.sql"
  }

  output = {
    format   = "<format>"
    schema   = "schema/<name_of_the_schema>.schema.json"
  }
}

Additionally, it's necessary to configure credentials for AWS. This can be done in a separate file terraform.tfvars with the following content:

aws = {
  region = "REGION"
  access_key = "ACCESS_KEY"
  secret_key = "SECRET_KEY"
}

The specified credentials can then be referenced and forwarded in the other *.tf files.

The module creates an RESTful endpoint via AWS lambda (e.g. https://3jopsshxxc.execute-api.eu-central-1.amazonaws.com/prod/). This endpoint can be used as an input for another data product or to retrieve information about this data product.

{
  "domain": "<data_product_domain>",
  "name": "<data_product_name>",
  "output": {
    "location": "arn:aws:s3:::<s3_bucket_name>/output/data/"
  }
}

See examples repository.

This terraform module is maintained by André Deuerling, Jochen Christ, and Simon Harrer.

MIT License.

No modules.

object({
    region     = string
    access_key = string
    secret_key = string
  })

list(object({
    source     = string
  }))

object({
    format     = string
    schema     = string
  })

object({
    query = string
  })

No outputs.