joona / aws-es-curl Goto Github PK

I always receive the following error:

{"message":"Credential should be scoped to a valid region, not 'eu-west-1'. "}

I've confirmed that my aws cli is configured correctly:

$ aws sts get-caller-identity --profile <profile name>
{
    "UserId": "",
    "Account": "",
    "Arn": "arn:aws:iam:::user/"
}
$ aws-es-curl --profile <profile name> -X GET https://<endpoint>.<region>.es.amazonaws.com
{"message":"Credential should be scoped to a valid region, not 'eu-west-1'. "}
$

Has anyone else seen this?

Hello, I get this error on running:

[ec2-user@ip-xxx-xx-xx-xx ~]$ aws-es-curl 

/usr/lib/node_modules/aws-es-curl/index.js:14
var getCredentials = function*() {
                             ^
SyntaxError: Unexpected token *
    at Module._compile (module.js:439:25)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/usr/lib/node_modules/aws-es-curl/bin/aws-es-curl:2:14)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)

Tested with an instance profile and adding the IAM role to ~/.aws/credentials;
aws-es-curl --region us-east-1 -X GET es-url/_search returns no output.
What additional logs can I provide?

Using the default install of aws-es-curl, and attempting to POST to /_bulk will result in the following error:
{ "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "The bulk request must be terminated by a newline [\\n]" } ], "type": "illegal_argument_exception", "reason": "The bulk request must be terminated by a newline [\\n]" }, "status": 400 }

Modifying this line resolve(data.trim()); to remove the .trim() will get you past this error.

On my system, this file was located here after running npm install aws-es-curl -g
/opt/nvm/versions/node/v10.16.0/lib/node_modules/aws-es-curl/index.js

The command used was
cat stats.json | aws-es-curl -X POST "https://search-redacted.es.amazonaws.com/_bulk".

As bulk requests are multiple line, it would be nice to allow for the following:
aws-es-curl -X POST "https://search-redacted.es.amazonaws.com/_bulk" --data @stats.json

nodejs v4 is required, which is not stated in the readme, nor in the package.json file.

#5

Hi Joona,

first, thanks for this cool little tool wich saved my relationship with AWS-ES :-)

The only thing i am not aware of are, that the output is cut off sometimes.
Here is a little example:

aws-es-curl --profile daily --region eu-central-1 "http://DOMAIN/_cat/recovery?v&h=i,s,t,ty,st,rep,snap,f,fp,b,bp"

the output:

grld-test-42659accountingjournalmeta  2 250ms snapshot    done  s3snapshots-daily-accountingjournal snapshot6 4  100.0% 6771   100.0%
grld-test-42659accountingjournalmeta  3 114ms snapshot    done  s3snaps[myprompt@macbook]

No big deal i think.

hartmut

I spent a few hours trying to debug why my index mappings weren't sticking. Turns out it's because the only way to get body info into the tool is via pipe, not the -d that is in all the ES docs, and even when I did that, the -H flags weren't accepted and therefore any json documents were rejected because of wrong encoding.

Hi there,

today i got this error:

Error: Key must be a buffer
TypeError: Key must be a buffer
at new Hmac (crypto.js:115:16)
at Object.Hmac (crypto.js:113:12)
at Object.hmac (/usr/local/lib/node_modules/aws-es-curl/node_modules/aws-sdk/lib/util.js:401:30)
at Object.getSigningKey (/usr/local/lib/node_modules/aws-es-curl/node_modules/aws-sdk/lib/signers/v4_credentials.js:59:8)
at V4.signature (/usr/local/lib/node_modules/aws-es-curl/node_modules/aws-sdk/lib/signers/v4.js:97:36)
at V4.authorization (/usr/local/lib/node_modules/aws-es-curl/node_modules/aws-sdk/lib/signers/v4.js:92:36)
at V4.addAuthorization (/usr/local/lib/node_modules/aws-es-curl/node_modules/aws-sdk/lib/signers/v4.js:34:12)
at Promise (/usr/local/lib/node_modules/aws-es-curl/index.js:76:12)
at Promise ()
at execute (/usr/local/lib/node_modules/aws-es-curl/index.js:58:10)

INFO:
$ node -v
v8.6.0

aws --version
aws-cli/1.11.86 Python/2.7.10 Darwin/17.3.0 botocore/1.5.83

any hint?

Thanks folks.

Hartmut

First, thank you this lib is great.

I was able to use it to interact with my es domain, create indices, add documents, etc.

But when I try to send a search query it fails each time with:

{
  "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."
}

Query:

echo '{"query": {"title": "atl"}}' | aws-es-curl -X GET https://<domain>.us-east-1.es.amazonaws.com/demo-test/_search | jq

Am I doing something wrong?

While running the query i am getting the following error. i could see the index by doing he GET request. but i cannot delete the index

aws-es-curl --region us-east-1 -XDELETE 'https://myelk-vul5rrfxxxixxi.us-east-1.es.amazonaws.com/filebeat-2018.04.22'

Unable to access metadata service. connect EHOSTUNREACH 169.254.169.254:80
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"filebeat-2018.04.22","index_uuid":"na","index":"filebeat-2018.04.22"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"filebeat-2018.04.22","index_uuid":"na","index":"filebeat-2018.04.22"},"status":404

I'm trying to use aws-es-curl inside a kubernetes container (jenkins-agent) and noticed the provided serviceaccount role that should authenticate using OIDC webprovider is ignored. Making it default to the EC2 instance profile role.
I have seen this in our own Java code when the aws sdk used, was too old. I would really love to see a aws-es-curl updated to a newer sdk. (probably Node >= 2.521.0 sdk ) https://medium.com/@marcincuber/amazon-eks-with-oidc-provider-iam-roles-for-kubernetes-services-accounts-59015d15cb0c