Giter Club home page Giter Club logo

hsiehshujeng / cdk-emrserverless-with-delta-lake Goto Github PK

View Code? Open in Web Editor NEW
8.0 3.0 4.0 34.41 MB

This construct builds some elements for you to quickly launch an EMR Serverless application. After submitting the Emr Serverless job, you could also launch an EMR notebook via cluster template to check the outcome from the EMR Serverless application.

License: Apache License 2.0

JavaScript 8.62% TypeScript 76.23% Jupyter Notebook 11.60% Python 3.40% Shell 0.16%
cdk-constructs delta-lake dotnet emr-serverless java javacript projen python aws-service-catalog emr-notebooks

cdk-emrserverless-with-delta-lake's Introduction

cdk-emrserverless-with-delta-lake

License Release npm downloads pypi downloads NuGet downlods repo languages

npm (JS/TS) PyPI (Python) Maven (Java) Go NuGet
Link Link Link Link Link

high level architecture

This constrcut builds an EMR studio, a cluster template for the EMR Studio, and an EMR Serverless application. 2 S3 buckets will be created, one is for the EMR Studio workspace and the other one is for EMR Serverless applications. Besides, the VPC and the subnets for the EMR Studio will be tagged {"Key": "for-use-with-amazon-emr-managed-policies", "Value": "true"} via a custom resource. This is necessary for the service role of EMR Studio.
This construct is for analysts, data engineers, and anyone who wants to know how to process Delta Lake data with EMR serverless.
cfn designer
They build the construct via cdkv2 and build a serverless job within the EMR application generated by the construct via AWS CLI within few minutes. After the EMR serverless job is finished, they can then check the processed result done by the EMR serverless job on an EMR notebook through the cluster template.
app history

TOC

Requirements

  1. Your current identity has the AdministratorAccess power.
  2. An IAM user named Administrator with the AdministratorAccess power.
    • This is related to the Portfolio of AWS Service Catalog created by the construct, which is required for EMR cluster tempaltes.
    • You can choose whatsoever identity you wish to associate with the Product in the Porfolio for creating an EMR cluster via cluster tempalte. Check serviceCatalogProps in the EmrServerless construct for detail, otherwise, the IAM user mentioned above will be chosen to set up with the Product.

Before deployment

You might want to execute the following command.

PROFILE_NAME="scott.hsieh"
# If you only have one credentials on your local machine, just ignore `--profile`, buddy.  
cdk bootstrap aws://${AWS_ACCOUNT_ID}/${AWS_REGION} --profile ${PROFILE_NAME}

Minimal content for deployment

#!/usr/bin/env node
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { EmrServerless } from 'cdk-emrserverless-with-delta-lake';

class TypescriptStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    new EmrServerless(this, 'EmrServerless');
  }
}

const app = new cdk.App();
new TypescriptStack(app, 'TypescriptStack', {
  stackName: 'emr-studio',
  env: {
    region: process.env.CDK_DEFAULT_REGION,
    account: process.env.CDK_DEFAULT_ACCOUNT,
  },
});

After deployment

Promise me, darling, make advantage on the CloudFormation outputs. All you need is copy-paste, copy-paste, copy-paste, life should be always that easy.
cfn outputs

  1. Define the following environment variables on your current session.
    export PROFILE_NAME="${YOUR_PROFILE_NAME}"
    export JOB_ROLE_ARN="${copy-paste-thank-you}"
    export APPLICATION_ID="${copy-paste-thank-you}"
    export SERVERLESS_BUCKET_NAME="${copy-paste-thank-you}"
    export DELTA_LAKE_SCRIPT_NAME="delta-lake-demo"
    
  2. Copy partial NYC-taxi data into the EMR Serverless bucket.
    aws s3 cp s3://nyc-tlc/trip\ data/ s3://${SERVERLESS_BUCKET_NAME}/nyc-taxi/ --exclude "*" --include "yellow_tripdata_2021-*.parquet" --recursive --profile ${PROFILE_NAME}
  3. Create a Python script for processing Delta Lake
    touch ${DELTA_LAKE_SCRIPT_NAME}.py
    cat << EOF > ${DELTA_LAKE_SCRIPT_NAME}.py
    from pyspark.sql import SparkSession
    import uuid
    
    if __name__ == "__main__":
        """
            Delta Lake with EMR Serverless, take NYC taxi as example.
        """
        spark = SparkSession \\
            .builder \\
            .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \\
            .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \\
            .enableHiveSupport() \\
            .appName("Delta-Lake-OSS") \\
            .getOrCreate()
    
        url = "s3://${SERVERLESS_BUCKET_NAME}/emr-serverless-spark/delta-lake/output/1.2.1/%s/" % str(
            uuid.uuid4())
    
        # creates a Delta table and outputs to target S3 bucket
        spark.range(5).write.format("delta").save(url)
    
        # reads a Delta table and outputs to target S3 bucket
        spark.read.format("delta").load(url).show()
    
        # The source for the second Delta table.
        base = spark.read.parquet(
            "s3://${SERVERLESS_BUCKET_NAME}/nyc-taxi/*.parquet")
    
        # The sceond Delta table, oh ya.
        base.write.format("delta") \\
            .mode("overwrite") \\
            .save("s3://${SERVERLESS_BUCKET_NAME}/emr-serverless-spark/delta-lake/nyx-tlc-2021")
        spark.stop()
    EOF
  4. Upload the script and required jars into the serverless bucket
    # upload script
    aws s3 cp delta-lake-demo.py s3://${SERVERLESS_BUCKET_NAME}/scripts/${DELTA_LAKE_SCRIPT_NAME}.py --profile ${PROFILE_NAME}
    # download jars and upload them
    DELTA_VERSION="2.2.0"
    DELTA_LAKE_CORE="delta-core_2.13-${DELTA_VERSION}.jar"
    DELTA_LAKE_STORAGE="delta-storage-${DELTA_VERSION}.jar"
    curl https://repo1.maven.org/maven2/io/delta/delta-core_2.13/${DELTA_VERSION}/${DELTA_LAKE_CORE} --output ${DELTA_LAKE_CORE}
    curl https://repo1.maven.org/maven2/io/delta/delta-storage/${DELTA_VERSION}/${DELTA_LAKE_STORAGE} --output ${DELTA_LAKE_STORAGE}
    aws s3 mv ${DELTA_LAKE_CORE} s3://${SERVERLESS_BUCKET_NAME}/jars/${${DELTA_LAKE_CORE}} --profile ${PROFILE_NAME}
    aws s3 mv ${DELTA_LAKE_STORAGE} s3://${SERVERLESS_BUCKET_NAME}/jars/${DELTA_LAKE_STORAGE} --profile ${PROFILE_NAME}

Create an EMR Serverless app

Rememeber, you got so much information to copy and paste from the CloudFormation outputs.
cfn outputs

aws emr-serverless start-job-run \
  --application-id ${APPLICATION_ID} \
  --execution-role-arn ${JOB_ROLE_ARN} \
  --name 'shy-shy-first-time' \
  --job-driver '{
        "sparkSubmit": {
            "entryPoint": "s3://'${SERVERLESS_BUCKET_NAME}'/scripts/'${DELTA_LAKE_SCRIPT_NAME}'.py",
            "sparkSubmitParameters": "--conf spark.executor.cores=1 --conf spark.executor.memory=4g --conf spark.driver.cores=1 --conf spark.driver.memory=4g --conf spark.executor.instances=1 --conf spark.jars=s3://'${SERVERLESS_BUCKET_NAME}'/jars/delta-core_2.12-1.2.0.jar,s3://'${SERVERLESS_BUCKET_NAME}'/jars/delta-storage-1.2.0.jar"
        }
    }' \
  --configuration-overrides '{
        "monitoringConfiguration": {
            "s3MonitoringConfiguration": {
                "logUri": "s3://'${SERVERLESS_BUCKET_NAME}'/serverless-log/"
	        }
	    }
	}' \
	--profile ${PROFILE_NAME}

If you execute with success, you should see similar reponse as the following:

{
    "applicationId": "00f1gvklchoqru25",
    "jobRunId": "00f1h0ipd2maem01",
    "arn": "arn:aws:emr-serverless:ap-northeast-1:630778274080:/applications/00f1gvklchoqru25/jobruns/00f1h0ipd2maem01"
}

and got a Delta Lake data under s3://${SERVERLESS_BUCKET_NAME}/emr-serverless-spark/delta-lake/nyx-tlc-2021/.
Delta Lake data

Check the executing job

Access the EMR Studio via the URL from the CloudFormation outputs. It should look very similar to the following url: https://es-pilibalapilibala.emrstudio-prod.ap-northeast-1.amazonaws.com, i.e., weird string and region won't be the same as mine.

  1. Enter into the application
    enter into the app
  2. Enter into the executing job

Check results from an EMR notebook via cluster template

  1. Create a workspace and an EMR cluster via the cluster template on the AWS Console
    create workspace
  2. Check the results delivered by the EMR serverless application via an EMR notebook.

Fun facts

  1. You can assign multiple jars as a comma-separated list to the spark.jars as the Spark page says for your EMR Serverless job. The UI will complain, you still can start the job. Don't be afraid, just click it like when you were child, facing authority fearlessly.
    ui bug
  2. To fully delet a stack with the construct, you need to make sure there is no more workspace within the EMR Studio. Aside from that, you also need to remove the associated identity from the Service Catalog (this is a necessary resource for the cluster template).
  3. Version inconsistency on Spark history. Possibly it can be ignored yet still made me wonder why the versions are different.
    naughty inconsistency
  4. So far, I still haven't figured out how to make the s3a URI work. The s3 URI is fine while the serverless app will complain that it couldn't find proper credentials provider to read the s3a URI.

Future work

  1. Custom resuorce for EMR Serverless
  2. Make the construct more flexible for users
  3. Compare Databricks Runtime and EMR Serverless.

cdk-emrserverless-with-delta-lake's People

Contributors

hsiehshujeng avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

cdk-emrserverless-with-delta-lake's Issues

error TS2709: Cannot use namespace 'ResponseLike' as a type.

Describe the bug

Since Sep. 3rd, 2022 at 9:24 AM GTM+8, the automatic upgrading procedure would incur the following error message and thereafter the progress stops working as usual.

ode_modules/@types/cacheable-request/index.d.ts:26:42 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

26         cb?: (response: ServerResponse | ResponseLike) => void
                                            ~~~~~~~~~~~~
node_modules/@types/cacheable-request/index.d.ts:77:51 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

77             listener: (response: ServerResponse | ResponseLike) => void
                                                     ~~~~~~~~~~~~
node_modules/@types/cacheable-request/index.d.ts:81:69 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

81         on(event: 'response', listener: (response: ServerResponse | ResponseLike) => void): this;
                                                                       ~~~~~~~~~~~~
node_modules/@types/cacheable-request/index.d.ts:84:71 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

84         once(event: 'response', listener: (response: ServerResponse | ResponseLike) => void): this;
                                                                         ~~~~~~~~~~~~
node_modules/@types/cacheable-request/index.d.ts:89:51 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

89             listener: (response: ServerResponse | ResponseLike) => void
                                                     ~~~~~~~~~~~~
node_modules/@types/cacheable-request/index.d.ts:95:51 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

95             listener: (response: ServerResponse | ResponseLike) => void
                                                     ~~~~~~~~~~~~
node_modules/@types/cacheable-request/index.d.ts:104:51 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

104             listener: (response: ServerResponse | ResponseLike) => void
                                                      ~~~~~~~~~~~~
node_modules/@types/cacheable-request/index.d.ts:108:70 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

108         off(event: 'response', listener: (response: ServerResponse | ResponseLike) => void): this;
                                                                         ~~~~~~~~~~~~
node_modules/@types/cacheable-request/index.d.ts:112:73 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

112         listeners(event: 'response'): Array<(response: ServerResponse | ResponseLike) => void>;
                                                                            ~~~~~~~~~~~~
node_modules/@types/cacheable-request/index.d.ts:115:76 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

115         rawListeners(event: 'response'): Array<(response: ServerResponse | ResponseLike) => void>;
                                                                               ~~~~~~~~~~~~
node_modules/@types/cacheable-request/index.d.ts:118:60 - error TS2709: Cannot use namespace 'ResponseLike' as a type.

118         emit(event: 'response', response: ServerResponse | ResponseLike): boolean;

Expected behavior

Running without failure in the automatic procedure and updating this construct package on daily basis.

Current behavior

Being executed in the automatic procedure and being executed in local machine would incur the above exception message.

Reproduction steps

Just execute projen upgrade and project build with order , baby.

Solution

Add project.package.addPackageResolutions('[email protected]'); into your .projenrc.js. For detailed explanation, see the following comment.

Additional information/context

projen version

0.61.41

CDK CLI Version

2.40.0

Node.js Version

18.2.0

OS

Local machine
ProductName: macOS
ProductVersion: 12.4
BuildVersion: 21F79
Github Actions
ubuntu-latest

Language

Typescript

Language Version

4.7.2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.