awslabs / serverless-photo-recognition Goto Github PK

A collection of 3 lambda functions that are invoked by Amazon S3 or Amazon API Gateway to analyze uploaded images with Amazon Rekognition and save picture labels to ElasticSearch (written in Kotlin)

Home Page: https://aws.amazon.com/blogs/ai/use-amazon-rekognition-to-build-an-end-to-end-serverless-photo-recognition-system/

License: Apache License 2.0

Kotlin 50.17% Shell 49.83%

amazon-cognito amazon-api-gateway amazon-rekognition lambda-functions aws-lambda kotlin elasticsearch serverless-photo-recognition jwt architecture

serverless-photo-recognition's Introduction

Author: Vladimir Budilov

You can watch the video explanation of this architecture here

Serverless Photo Recognition

What is it?

A collection of 3 lambda functions that are invoked by Amazon S3, Amazon API Gateway, and directly (RESTful calls) to analyze uploaded images in S3 with Amazon Rekognition and save picture metadata to ElasticSearch

Before you start

When you run the setup script your account will start incurring AWS infrastructure costs.
This project was written a while back, before CloudFormation was available for all of the services in use, hence my use of the AWS CLI. If I were to write this today I would use CDK to create all of the resources (from IAM roles to ElasticSearch service). Check out this project for a reference.

The Architecture

Adding an image

Removing an image

Searching images

Tech Stack

Required Tools

AWS Services Involved In This Architecture

AWS Setup

Before we start, let’s make sure that your working environment is setup to run the script. Here's what you'll need:

An AWS Account with a default VPC
Java 8
The latest AWS CLI (Tested with aws-cli/1.11.29 Python/2.7.12)
Linux or Mac OS to run the setup script (the setup script won't work on Windows)

The following command will setup all of the needed resources, as well as print out the sample command that you can run to test your configuration:

# Clone it from github
git clone --depth 1 https://github.com/awslabs/serverless-photo-recognition.git

# Run the installation script
cd serverless-photo-recognition/setup
./setupEnvironment.sh

Running the installation script will modify the /src/main/kotlin/com/budilov/Properties.kt file with the newly-generated values. If you need to rerun the script, just revert the changes (otherwise the script will fail).

It will also create a deletion script under the tmp directory

At the end of the script, you'll see 3 commands (customized with your environment variables) that you can run to test out your configuration. Run the commands and see the functionality in action (copying and running the below commands will fail...use the ones that the script has generated):

-> Upload a picture
--------------------
aws s3 cp new-york.jpg s3://rekognition-20170307160632/usercontent/us-east-1:c61126b8-7f7b-48e8-8534-4c3a21dfef4e/

-> Sample search command
You might need to pipe the output to 'native2ascii -encoding UTF-8 -reverse' if you
want to copy and paste the signed url in the browser since curl encodes url output
-------------------------
curl -X POST -H "Authorization: $(aws cognito-idp admin-initiate-auth  --user-pool-id us-east-1_AEzYFK4mc --client-id 734810igh3bfdj4n33tfm9o08s --cli-input-json file:///tmp/authflow.json --query AuthenticationResult.IdToken --output text)" -H "search-key: building" -H "Cache-Control: no-cache" "https://57zt8cwa6j.execute-api.us-east-1.amazonaws.com/prod/picture/search/"

-> Remove the picture
----------------------
aws s3 rm s3://rekognition-20170307160632/usercontent/us-east-1:c61126b8-7f7b-48e8-8534-4c3a21dfef4e/new-york.jpg

serverless-photo-recognition's People

Contributors

Stargazers

Watchers

Forkers

belminf cmanikandan pierreliddle adhorn hazer tomhoag chalam petergrigoriev redavis22 stephensalim gronau-it-cloud-computing radityagumay tarun87 yaareck vikitripathi lunchact grelaxus milrzeit ajurasz abhinav-cleo dharmakshetri sergeydolgov1 mitchell-accedo imtepiny erkapilmehta gaybro8777 sportsbitenews pikamuchu wchoy jinqian monkeyer cdevops dy604 vicky002 xdslab raniemi gridl jendave pbota athiwatp coreysalzer-amazon juanlp gopicares hanifmahboobi louth spencerx neo4reo jretza totallychefs vineet2garg ssh-shashi nguyendat junaidpk daitc2004 julien-abt waseemarain4u wanglaibao rajgit2012 mikemac42 lastmanstandingv2 orf53975 dwtcourses baiyongzhen avishekdas multi-cloud-training narinkaew bracomitg brac10 ik-serverless low205 keeplearningandtrying ihao8 igorjs amitkalawat nitinls prashanthjs86 frankswu habibtalib tubbz-alt ninjayoto nobi-attapol reayd-chetwood anibus centsten2000 chengjingfeng michaelrishiforrester jipengxiang simhaonline arnab2203 alexrogalskiy michaelclapham michaelanis14 musa-abraham juniorbi rohitx18 brahimmade vamsi-desireddy yashprajapati622 saketaram0209 stevenoyar

serverless-photo-recognition's Issues

some erros with setup script

I am running this on Debian, java8, python 2.7 and awscli installed... It outputs some erros and then goes on forever with "Waiting for the ES domain rekognition20190124114720 to finish processing. Sleeping.."

./setupEnvironment.sh
Using the following 'root name' for your resources, such as S3 bucket: 20190124114720
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 15 100 15 0 0 45 0 --:--:-- --:--:-- --:--:-- 45
Creating the Cognito resources
{
"Role": {
"Arn": "arn:aws:iam::179810983012:role/20190124114720rek-unauthenticated-role",
"AssumeRolePolicyDocument": {
"Statement": [
{
"Condition": {
"ForAnyValue:StringLike": {
"cognito-identity.amazonaws.com:amr": "unauthenticated"
},
"StringEquals": {
"cognito-identity.amazonaws.com:aud": "us-east-1:9be46eae-5a3d-4a18-8b38-9b9e4ff8803c"
}
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Principal": {
"Federated": "cognito-identity.amazonaws.com"
},
"Effect": "Allow"
}
],
"Version": "2012-10-17"
},
"RoleName": "20190124114720rek-unauthenticated-role",
"RoleId": "AROAJN66ZUIEFDLUYRREA",
"CreateDate": "2019-01-24T10:47:24Z",
"Path": "/"
}
}
{
"Role": {
"Arn": "arn:aws:iam::179810983012:role/20190124114720rek-authenticated-role",
"RoleId": "AROAJLGGGEFVRQ3E227SO",
"AssumeRolePolicyDocument": {
"Statement": [
{
"Condition": {
"ForAnyValue:StringLike": {
"cognito-identity.amazonaws.com:amr": "authenticated"
},
"StringEquals": {
"cognito-identity.amazonaws.com:aud": "us-east-1:9be46eae-5a3d-4a18-8b38-9b9e4ff8803c"
}
},
"Effect": "Allow",
"Principal": {
"Federated": "cognito-identity.amazonaws.com"
},
"Action": "sts:AssumeRoleWithWebIdentity"
}
],
"Version": "2012-10-17"
},
"CreateDate": "2019-01-24T10:47:27Z",
"Path": "/",
"RoleName": "20190124114720rek-authenticated-role"
}
}
usage: aws [options] [ ...] [parameters]
To see help text, you can run:

aws help
aws help
aws help

Unknown options: --schema, Name=email,Required=true
usage: aws [options] [ ...] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --user-pool-id: expected one argument

Parameter validation failed:
Invalid length for parameter CognitoIdentityProviders[0].ClientId, value: 0, valid range: 1-inf
{
"Role": {
"RoleName": "lambda-to-es-rek-s3-20190124114720",
"Arn": "arn:aws:iam::179810983012:role/lambda-to-es-rek-s3-20190124114720",
"RoleId": "AROAJTEJ7B6OPZMPIA3QA",
"Path": "/",
"CreateDate": "2019-01-24T10:47:34Z",
"AssumeRolePolicyDocument": {
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
}
}
],
"Version": "2012-10-17"
}
}
}
make_bucket: rekognition-20190124114720
The Task.leftShift(Closure) method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use Task.doLast(Action) instead.
at build_cr2xxzh92n7ciadgvde49npdd.run(/home/openhab/serverless-photo-recognition/build.gradle:48)
:compileKotlin
w: /home/openhab/serverless-photo-recognition/src/main/kotlin/com/budilov/db/ESPictureService.kt: (91, 23): 'getSourceAsObjectList(Class<T!>!): (Mutable)List<T!>!' is deprecated. Deprecated in Java
:compileJava UP-TO-DATE
:copyMainKotlinClasses
:processResources UP-TO-DATE
:classes UP-TO-DATE
:jar
:startScripts
:distTar
:distZip
:assemble
:compileTestKotlin UP-TO-DATE
:compileTestJava UP-TO-DATE
:copyTestKotlinClasses
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:test UP-TO-DATE
:check UP-TO-DATE
:build

BUILD SUCCESSFUL

Total time: 13.657 secs
Setting up the S3 Lambda events
Adding the ADD event Lambda permissions
{
"Statement": "{"Sid":"20190124114720-rekognition","Effect":"Allow","Principal":{"Service":"s3.amazonaws.com"},"Action":"lambda:InvokeFunction","Resource":"arn:aws:lambda:us-east-1:179810983012:function:rekognition-add-picture-20190124114720","Condition":{"StringEquals":{"AWS:SourceAccount":"179810983012"},"ArnLike":{"AWS:SourceArn":"arn:aws:s3:::rekognition-20190124114720"}}}"
}
Adding the DEL event Lambda permissions
{
"Statement": "{"Sid":"20190124114720-rekognition","Effect":"Allow","Principal":{"Service":"s3.amazonaws.com"},"Action":"lambda:InvokeFunction","Resource":"arn:aws:lambda:us-east-1:179810983012:function:rekognition-del-picture-20190124114720","Condition":{"StringEquals":{"AWS:SourceAccount":"179810983012"},"ArnLike":{"AWS:SourceArn":"arn:aws:s3:::rekognition-20190124114720"}}}"
}
Preparing the S3 Notifications json definitions file
Creating the events
Setup Elasticsearch with a domain of rekognition20190124114720
{
"DomainStatus": {
"AdvancedOptions": {
"rest.action.multi.allow_explicit_index": "true"
},
"DomainId": "179810983012/rekognition20190124114720",
"Deleted": false,
"DomainName": "rekognition20190124114720",
"SnapshotOptions": {
"AutomatedSnapshotStartHour": 0
},
"AccessPolicies": "",
"Processing": true,
"EBSOptions": {
"VolumeType": "standard",
"VolumeSize": 10,
"EBSEnabled": true
},
"ElasticsearchVersion": "5.1",
"ARN": "arn:aws:es:us-east-1:179810983012:domain/rekognition20190124114720",
"Created": true,
"ElasticsearchClusterConfig": {
"InstanceCount": 1,
"InstanceType": "t2.small.elasticsearch",
"DedicatedMasterEnabled": false,
"ZoneAwarenessEnabled": false
}
}
}
{
"DomainConfig": {
"ElasticsearchVersion": {
"Status": {
"State": "Processing",
"UpdateDate": 1548326914.886,
"PendingDeletion": false,
"UpdateVersion": 2,
"CreationDate": 1548326914.886
},
"Options": "5.1"
},
"ElasticsearchClusterConfig": {
"Status": {
"State": "Processing",
"UpdateDate": 1548326914.886,
"PendingDeletion": false,
"UpdateVersion": 2,
"CreationDate": 1548326914.886
},
"Options": {
"InstanceCount": 1,
"InstanceType": "t2.small.elasticsearch",
"ZoneAwarenessEnabled": false,
"DedicatedMasterEnabled": false
}
},
"AccessPolicies": {
"Status": {
"State": "Processing",
"UpdateDate": 1548326918.297,
"PendingDeletion": false,
"UpdateVersion": 4,
"CreationDate": 1548326918.297
},
"Options": "{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::179810983012:role/lambda-to-es-rek-s3-20190124114720"},"Action":"es:","Resource":"arn:aws:es:us-east-1:179810983012:domain/rekognition20190124114720/"},{"Sid":"","Effect":"Allow","Principal":{"AWS":""},"Action":"es:","Resource":"arn:aws:es:us-east-1:179810983012:domain/rekognition20190124114720/*","Condition":{"IpAddress":{"aws:SourceIp":"84.255.242.202"}}}]}"
},
"EBSOptions": {
"Status": {
"State": "Processing",
"UpdateDate": 1548326914.886,
"PendingDeletion": false,
"UpdateVersion": 2,
"CreationDate": 1548326914.886
},
"Options": {
"VolumeSize": 10,
"VolumeType": "standard",
"EBSEnabled": true
}
},
"SnapshotOptions": {
"Status": {
"State": "Processing",
"UpdateDate": 1548326914.886,
"PendingDeletion": false,
"UpdateVersion": 2,
"CreationDate": 1548326914.886
},
"Options": {
"AutomatedSnapshotStartHour": 0
}
},
"AdvancedOptions": {
"Status": {
"State": "Active",
"UpdateDate": 1548326918.47,
"PendingDeletion": false,
"UpdateVersion": 4,
"CreationDate": 1548326918.47
},
"Options": {
"rest.action.multi.allow_explicit_index": "true"
}
}
}
}
Waiting for the ES domain rekognition20190124114720 to finish processing. Sleeping...
Waiting for the ES domain rekognition20190124114720 to finish processing. Sleeping..
Waiting for the ES domain rekognition20190124114720 to finish processing. Sleeping...
Waiting for the ES domain rekognition20190124114720 to finish processing. Sleeping...

While running ./setupEnvironment.sh gives error

After getting code to my local, i tried to run the command..i am getting this error.. i have installed aws client and awslogs..do i need to install anything else ?
D:\aPaaS\Aws\projects\serverless-photo-recognition-master\setup>./setupEnvironment.sh
'.' is not recognized as an internal or external command,

API Gateway creation error

Hi,

I was trying to run the script and even when the shell shows "You're done" actually it get stuck on this script:

echo "Importing the swagger template"
aws apigateway import-rest-api --body 'file:///tmp/apigateway-swagger.json' --region ${REGION} > /tmp/apigateway-import-api  --region ${REGION}
GATEWAY_ID=$(cat /tmp/apigateway-import-api | grep id | awk '{print $2}' | xargs |sed -e 's/^"//'  -e 's/"$//' -e 's/,$//')

I believe the problem is aws apigateway becuase in shell returns "Invalid base64".
On the documentation it seems like I need to add --cli-binary-format raw-in-base64-out but even with that I don't get the response to create the tmp file.

Cannot perform search

I'm not able to successfully query for a label. I've got my cognito id and id token from the cognito website. The example curl command with all the proper substitutions returns {"message":"Unauthorized"}

If I test in the API Gateway console setting the headers with the appropriate values for search-key, Authorization and Cache-Control , I get a result of { "message": "Internal server error" } with a very long log entry that I've pasted in below. The best indicator of something wrong is the Execution failed due to configuration error: Invalid permissions on Lambda function but I cannot find anything missing from the role. The lambda search function does not create any log entry.

I have API Gateway logging to Cloudwatch, but those log entries aren't very helpful:

Starting authorizer: q1rara for request: eecf6fe8-fb0c-11e6-bedc-a9d3c9017e82
Unauthorized request: eecf6fe8-fb0c-11e6-bedc-a9d3c9017e82

Any ideas on where to look next??

Thanks

tom

Log message from API Gateway console test:
Execution log for request test-request
Sat Feb 25 03:48:37 UTC 2017 : Starting execution for request: test-invoke-request
Sat Feb 25 03:48:37 UTC 2017 : HTTP Method: POST, Resource Path: /picture/search
Sat Feb 25 03:48:37 UTC 2017 : Method request path: {}
Sat Feb 25 03:48:37 UTC 2017 : Method request query string: {}
Sat Feb 25 03:48:37 UTC 2017 : Method request headers: {Authorization=********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************zk_IEw, search-ke [TRUNCATED]
Sat Feb 25 03:48:37 UTC 2017 : Method request body before transformations:
Sat Feb 25 03:48:37 UTC 2017 : Endpoint request URI: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-east-1:266942040790:function:rekognition-search-picture-20170224214220/invocations
Sat Feb 25 03:48:37 UTC 2017 : Endpoint request headers: {x-amzn-lambda-integration-tag=test-request, Authorization=****************************************************************************************************************************************************************************************************************************************************************************************************************************************3a677f, X-Amz-Date=20170225T034837Z, x-amzn-apigateway-api-id=qq4l1zipaf, X-Amz-Source-Arn=arn:aws:execute-api:us-east-1:266942040790:qq4l1zipaf/null/POST/picture/search, Accept=application/json, User-Agent=AmazonAPIGateway_qq4l1zipaf, X-Amz-Security-Token=FQoDYXdzEPz//////////wEaDHmbwkC/YeFe+3lMGSK3A46e9Gdt+IISFWcZO2JxmcKM1sl7OrfYXUStcUxVxsFHCRqqbT/gWgs/Cgr2soaWw2oNJqmDAvcN5WclNlz811AWdKrp3Ka6hsLmpuD4TlnGrNFk28PadvS9k719NyLSFpp5OzXRL6kACu9BRsCUxPz4bC90bDaDnybfrmYpQFDfwqTDHM84P4gG64efVLw6T0IAYmr4lFnbozKd2F77c2nCyS+CERQRHA+L6wpYr+sPwDaDSMs9AfLhMtjQOUrHMgdy4BTXS7fnF0dqVditYrUV5/IwGNvV4ZqwRu08O0cQ5glcM028Eu [TRUNCATED]
Sat Feb 25 03:48:37 UTC 2017 : Endpoint request body after transformations: {"resource":"/picture/search","path":"/picture/search","httpMethod":"POST","headers":{"Authorization":"eyJraWQiOiJuVDBDSVBRcVl1OWo5d1JZZHp3SGxZRkZkaU83ZWtEQUQwQVpFUUJOeFJnPSIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJhMTAyNDVjMy1mMDdlLTQwOTctYTJkZS1mMjM1ODYwNWVkZjQiLCJkZXZpY2Vfa2V5IjoidXMtZWFzdC0xXzJiYmY3NTIxLTJiMTYtNDEwYS04MTBhLTgyYzFlYmU2NGE4YSIsInRva2VuX3VzZSI6ImFjY2VzcyIsInNjb3BlIjoiYXdzLmNvZ25pdG8uc2lnbmluLnVzZXIuYWRtaW4iLCJpc3MiOiJodHRwczpcL1wvY29nbml0by1pZHAudXMtZWFzdC0xLmFtYXpvbmF3cy5jb21cL3VzLWVhc3QtMV9QR1NiQ1ZaN1MiLCJleHAiOjE0ODc5OTU3MjQsImlhdCI6MTQ4Nzk5MjEyNCwianRpIjoiNjExNWZiYTctODc0OC00ZDg4LWI0MGUtNGYyNTIyNWMwYzM0IiwiY2xpZW50X2lkIjoiaGg1aWJ2NjdzbzBxdWt0NTVjNXVsYWx0ayIsInVzZXJuYW1lIjoidG9taG9hZytmb3VyQGdtYWlsLmNvbSJ9.CFImt7N47-oweyE1oJMzlB6QA13149SmUfgjkEpa-RHL672dWjkCFhyOKl9YnKM5M7_cw8ygVr-aF9ATG1INfFnCkFaj56i5VdgWsFPuo8xpthxZVb5jXf0EPkcKeRY-j-7ZvcUr8qzFk83zKTpYqwHiglnZZqK_oijB3P8DLTIjjWOXHhUGvoWOrQ8KlPUlXkzm9BWfjfQTlcdUMx01-1Fomw6tHbBP-hTjGhmyRSugIqKJAo5Z [TRUNCATED]
Sat Feb 25 03:48:37 UTC 2017 : Execution failed due to configuration error: Invalid permissions on Lambda function
Sat Feb 25 03:48:37 UTC 2017 : Method completed with status: 500

Search fails when launched in EU-West-1

Great tool, thanks!

Search fails (Sample command) because the Cognito pool apparently can't be found.

According to this: http://stackoverflow.com/questions/27137299/amazon-cognito-connection-cannot-find-identitypool may be a region issue.

Detailed trace:
IdentityPool 'eu-west-1:9759c8b7-2f08-4689-be3f-0fcb416aa091' not found. (Service: AmazonCognitoIdentity; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 06e74c12-0f3e-11e7-b97a-697f028f5f61): com.amazonaws.services.cognitoidentity.model.ResourceNotFoundException com.amazonaws.services.cognitoidentity.model.ResourceNotFoundException: IdentityPool 'eu-west-1:9759c8b7-2f08-4689-be3f-0fcb416aa091' not found. (Service: AmazonCognitoIdentity; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 06e74c12-0f3e-11e7-b97a-697f028f5f61) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1545) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1183) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447) at com.amazonaws.services.cognitoidentity.AmazonCognitoIdentityClient.doInvoke(AmazonCognitoIdentityClient.java:1385) at com.amazonaws.services.cognitoidentity.AmazonCognitoIdentityClient.invoke(AmazonCognitoIdentityClient.java:1361) at com.amazonaws.services.cognitoidentity.AmazonCognitoIdentityClient.getId(AmazonCognitoIdentityClient.java:679) at com.budilov.SearchPhotosHandler.getCognitoId(SearchPhotosHandler.kt:86) at com.budilov.SearchPhotosHandler.handleRequest(SearchPhotosHandler.kt:55) at com.budilov.SearchPhotosHandler.handleRequest(SearchPhotosHandler.kt:27)

Use of sed -i in setupEnvironment.sh

Hi Vladimir

When I run setupEnvironment.sh on OS X (10.12.3), the script errors out when attempting to sed the values into Properties.kt

sed: 1: "../src/main/kotlin/com/ ...": invalid command code .
sed: 1: "../src/main/kotlin/com/ ...": invalid command code .
sed: 1: "../src/main/kotlin/com/ ...": invalid command code .
sed: 1: "../src/main/kotlin/com/ ...": invalid command code .
sed: 1: "../src/main/kotlin/com/ ...": invalid command code .
sed: 1: "../src/main/kotlin/com/ ...": invalid command code .

I think this is because sed on OS X expects there to be a file extension immediately after the -i

instead of

sed -i 's#REGION_REPLACE_ME#'${REGION}'#g' ../src/main/kotlin/com/budilov/Properties.kt
OSX needs

sed -i.tmp 's#REGION_REPLACE_ME#'${REGION}'#g' ../src/main/kotlin/com/budilov/Properties.kt

Previously, the substitution was handled by piping all of the sed statements together -- not sure why that changed?

Anyway, I'm happy to make the change and provide a pull request, but I'm not sure what the behavior of the change will be on other OS's?

Update: changing -i to -i.tmp does fix the problem on OS X

Properties.kt is empty (or sometimes incomplete)

When running the setEnvironment.sh, sometimes the Properties.kt file ends up empty (i.e. file size = 0) and sometimes is is nearly complete but is missing a valid value for _ES_SERVICE_URL e.g.

val _ES_SERVICE_URL = "https://None"

I have not been able to track down the source of the file empty problem, but I suspect that populating the ES_SERVICE_URL is a timing issue?

ES_ENDPOINT=$(aws es describe-elasticsearch-domain --domain-name ${ES_DOMAIN_NAME} --query "DomainStatus.Endpoint" --output text)

is being executed immediately after the ES search domain is completed processing and is returning None. Maybe there is some intermediate state prior to the DomainStatus.Endpoint being populated??

Script Fails when output for CLI is not JSON

Found this out after some fumbling, I was set to text and that broke the SED/AWK etc. in the shell scripts.

Easy to deal with, but would be great to fix by forcing JSON output for CLI commands independently.

Add to awesome-functions

It's a good candidate for https://github.com/plutov/awesome-functions