The terraform-databricks-aws-workspace from sdsc-sherlock

Terraform module for Databricks AWS Workspace E2 (Part 1)

❗️ Important

👉 This Terraform module assumes you have access to: https://accounts.cloud.databricks.com

👉 Databricks account username: databricks_account_username

👉 Databricks account password: databricks_account_password

👉 Databricks account id, databricks_account_id can be found on the top right corner of the page, once you're logged in.

👉 Part 2: Terraform module for Databricks Workspace management

Versions

Module tested for Terraform 1.0.1.
databrickslabs/databricks provider version 0.3.5
AWS provider version 3.47.
main branch: Provider versions not pinned to keep up with Terraform releases.
tags releases: Tags are pinned with versions (use ).

Usage

Option 1:

terrafrom init
terraform plan -var='teamid=tryme' -var='prjid=project1'
terraform apply -var='teamid=tryme' -var='prjid=project1'
terraform destroy -var='teamid=tryme' -var='prjid=project1'

Note: With this option please take care of remote state storage

Option 2:

Recommended method (stores remote state in S3 using `prjid` and `teamid` to create directory structure):

Create python 3.6+ virtual environment

python3 -m venv <venv name>

Install package:

pip install tfremote --upgrade

Set below environment variables:

export TF_AWS_BUCKET=<remote state bucket name>
export TF_AWS_BUCKET_REGION=us-west-2
export TF_AWS_PROFILE=<profile from ~/.ws/credentials>

Set below environment variables:

export TF_AWS_BUCKET=<remote state bucket name>
export TF_AWS_BUCKET_REGION=us-west-2
export AWS_ACCESS_KEY_ID=<aws_access_key_id>
export AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>

Update main.tf file with required values.
Run and verify the output before deploying:

tf -c=aws plan -var='teamid=foo' -var='prjid=bar'

Run below to deploy:

tf -c=aws apply -var='teamid=foo' -var='prjid=bar'

Run below to destroy:

tf -c=aws destroy -var='teamid=foo' -var='prjid=bar'

NOTE:

Read more on tfremote

Databricks workspace creation with new role

module "databricks_workspace" {
  source = "git::[email protected]:tomarv2/terraform-databricks-aws-workspace.git"

  # NOTE: One of the below is required:
  # - 'profile_for_iam' - for IAM creation (if none is provided 'default' is used)
  # - 'existing_role_name'
  profile_for_iam             = "iam-admin"
  aws_region                  = "us-east-2"
  databricks_account_username = "[email protected]"
  databricks_account_password = "sample123!"
  databricks_account_id       = "1234567-1234-1234-1234-1234567"
  # -----------------------------------------
  # Do not change the teamid, prjid once set.
  teamid = var.teamid
  prjid  = var.prjid
}

Databricks workspace creation with existing role

module "databricks_workspace" {
  source = "git::[email protected]:tomarv2/terraform-databricks-aws-workspace.git"

  # NOTE: One of the below is required:
  # - 'profile_for_iam' - for IAM creation (if none is provided 'default' is used)
  # - 'existing_role_name'
  existing_role_arn          = "arn:aws:iam::123456789012:role/demo-role"
  aws_region                  = "us-east-2"
  databricks_account_username = "[email protected]"
  databricks_account_password = "sample123!"
  databricks_account_id       = "1234567-1234-1234-1234-1234567"
  # -----------------------------------------
  # Do not change the teamid, prjid once set.
  teamid = var.teamid
  prjid  = var.prjid
}

Please refer to examples directory link for references.

Coming up:

Use Customer Managed VPC

Troubleshooting:

IAM policy error

If you notice below error:

Error: MALFORMED_REQUEST: Failed credentials validation checks: Spot Cancellation, Create Placement Group, Delete Tags, Describe Availability Zones, Describe instances, Describe Instance Status, Describe Placement Group, Describe Route Tables, Describe Security Groups, Describe Spot Instances, Describe Spot Price History, Describe Subnets, Describe Volumes, Describe Vpcs, Request Spot Instances

Try creating workspace from UI:

Verify if the role and policy exists (assume role should allow external id)

Requirements

Name	Version
terraform	>= 1.0.1
aws	~> 3.47
databricks	0.3.5
random	~> 3.1
time	~> 0.7

Providers

Name	Version
aws	3.38.0
databricks	0.3.3
databricks.created_workspace	0.3.3
databricks.mws	0.3.3
random	3.1.0
time	0.7.1

Modules

Name	Source	Version
iam_policies	git::[email protected]:tomarv2/terraform-aws-iam-policies.git	v0.0.4
iam_role	git::[email protected]:tomarv2/terraform-aws-iam-role.git//modules/iam_role_external	v0.0.4
s3	git::[email protected]:tomarv2/terraform-aws-s3.git	v0.0.3
vpc	git::[email protected]:tomarv2/terraform-aws-vpc.git	v0.0.2

Resources

Name	Type
aws_s3_bucket_policy.root_bucket_policy	resource
databricks_mws_credentials.this	resource
databricks_mws_networks.this	resource
databricks_mws_storage_configurations.this	resource
databricks_mws_workspaces.this	resource
databricks_token.pat	resource
random_string.naming	resource
time_sleep.wait	resource
databricks_aws_assume_role_policy.this	data source
databricks_aws_bucket_policy.this	data source
databricks_aws_crossaccount_policy.cross_account_iam_policy	data source

Inputs

Name	Description	Type	Default	Required
aws_region	default aws region	`string`	`"us-west-2"`	no
databricks_account_id	External ID provided by third party.	`string`	n/a	yes
databricks_account_password	databricks account password	`string`	n/a	yes
databricks_account_username	databricks account username	`string`	n/a	yes
existing_role_name	If you want to use existing role name, else a new role will be created	`string`	`null`	no
prjid	(Required) Name of the project/stack e.g: mystack, nifieks, demoaci. Should not be changed after running 'tf apply'	`string`	n/a	yes
profile_for_iam	profile to use for IAM	`string`	`null`	no
profile_to_use	Getting values from ~/.aws/credentials	`string`	`"default"`	no
teamid	(Required) Name of the team/group e.g. devops, dataengineering. Should not be changed after running 'tf apply'	`string`	n/a	yes

Outputs

Name	Description
databricks_credentials_id	databricks credentials id
databricks_deployment_name	databricks deployment name
databricks_host	databricks hostname
databricks_mws_credentials_id	databricks mws credentials id
databricks_mws_network_id	databricks mws network id
databricks_mws_storage_bucket_name	databricks mws storage bucket name
databricks_mws_storage_id	databricks mws storage id
iam_role_arn	iam role arn
inline_policy_id	inline policy id
pat_token_duration	databricks pat
s3_bucket_arn	s3 bucket arn
s3_bucket_id	s3 bucket id
s3_bucket_name	s3 bucket name
storage_configuration_id	databricks storage configuration id
vpc_id	vpc id
workspace_url	databricks workspace url

sdsc-sherlock / terraform-databricks-aws-workspace Goto Github PK

terraform-databricks-aws-workspace's Introduction

Terraform module for Databricks AWS Workspace E2 (Part 1)

Versions

Usage

Option 1:

Option 2:

Recommended method (stores remote state in S3 using prjid and teamid to create directory structure):

Databricks workspace creation with new role

Databricks workspace creation with existing role

Coming up:

Troubleshooting:

IAM policy error

Requirements

Providers

Modules

Resources

Inputs

Outputs

terraform-databricks-aws-workspace's People

Contributors

Recommend Projects

Recommend Topics

Recommend Org

Recommended method (stores remote state in S3 using `prjid` and `teamid` to create directory structure):