This is a CDK project for streaming data analytics and visualize demonstration.

Following AWS services are deployed:

• Amazon Kinesis Data Stream
• Amazon Kinesis Data Firehose
• AWS Lambda
• Amazon OpenSearch Service
• Amazon Cognito

If you need, you can transform ingest data using Firehose transform Lambda.(See also Appendix:Transform Ingested Data)

Cognito is used for OpenSearch authentication and fine-grained access control is enabled.

• OpenSearch is deployed on VPC subnet, so to access you need reachability to vpc by reverse-proxy, VPN, or Direct Connect and so on.
• OpenSearch has fine-grained access control enabled.
• This project is for demonstration purposes only, not for production environment.
• Produced apps are not included. So please prepare the input data yourself.
• Deleting the stack also deletes the CloudWatch Logs.

The following resources must be created in advance and the resource IDs must be set in the Context Parameter in cdk.json or ~/.cdk.json. See the next section on "How to Deploy" for more information.

• VPC
• Subnet
• Security Group (for OpenSearch and Firehose)

• VPC
• Subnet
• Security Group (for OpenSearch and Firehose)

Of course, existing resources are also acceptable.

Configure cdk.json or ~/.cdk.json like following:

    context: {
        "kinesis-lambda-opensearch-stack": {
            "projectName": "demo-project",
            "vpcId": "vpc-0123456789",
            "openSearchDomainName": "domainName",
            "openSearchIndexName": "indexName",
            "securityGroupIdsForOpenSearch": [
                "sg-123456789abcdef01"
            ],
            "subnets": [
                {"subnetId": "subnet-0123456789abcdef0", "availabilityZone": "us-east-1a"}
            ],
            "bucketNameForFirehose": "bucketName",
            "securityGroupIdsForFirehose": [
                "sg-123456789abcdef01"
            ]
        }
    }

cdk deploy --all

Please wait to complete deployment. It will probably take more than 30 minutes.

Access the Cognito User Pool on the management console and create a master user.

Add the user to the Cognito User Pool Group named master, since the group has already been created with Master-User Role assigned.

With the user you created, you can now log in to your OpenSearch dashboard and access it!

Confirm that data submitted to Kinesis is stored in OpenSearch.

aws kinesis put-record \
  --stream-name ${STREAM_NAME} \
  --partition-key 1
  --data `echo -n '{"key":"message"}' | base64`

Confirm indices

GET /_cat/indices?v&s=index

Query index

GET /<INDEX_NAME>/_search
{
  "query": {
    "match_all": {}
  }
}

You can clean Up by following command:

cdk destroy --all

If you want to transform ingested data, you can use transform Lambda function. The function has already deployed. The source code is lambda/transform/app.py.

You modify the function code, then to deploy cdk deploy KinesisOpenSearch-TransformLambdaStack

MIT