Is this a request for help?: yes, or bug fix, not sure
Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug fix???
Version of Helm and Kubernetes:
helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
[ec2-user@ip-172-31-255-150 fluentd-elasticsearch]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-12-06T01:33:57Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.8-eks-7c34c0", GitCommit:"7c34c0d2f2d0f11f397d55a46945193a0e22d8f3", GitTreeState:"clean", BuildDate:"2019-03-01T22:49:39Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Which chart in which version:
fluentd-elasticsearch
version: 2.7.0
appVersion: 2.4.0
What happened:
can't connect to AWS Elasticsearch, suppress user/password if empty?
What you expected to happen:
connect to AWS Elasticsearch (6.4)
How to reproduce it (as minimally and precisely as possible):
helm install . --name fluentd-elasticsearch --namespace logging -f values-mine.yaml
< host: 'vpc-eks-logging-aaaaaaaaaaaaa.us-east-1.es.amazonaws.com'
1.es.amazonaws.com'
< port: 443
< scheme: 'https'
host: 'elasticsearch-client'
port: 9200
scheme: 'http'
39,40c37,40
user: ""
password: ""
Anything else we need to know:
[warn]: [elasticsearch] failed to flush the buffer. retry_time=5 next_retry_seconds=2019-03-27 22:13:38 +0000 chunk="58519368ae8d5ef06fb04c89b57fdcc7" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"vpc-eks-logging-6w44gqj2u5xkmsnic766dbezhm.us-east-1.es.amazonaws.com", :port=>443, :scheme=>"https", :user=>"", :password=>"obfuscated"}): [403] {"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. Authorization=Basic PG5pbD46PG5pbD4="}"
AWS support says...
it looks like your Fluentd is adding an authorization header to the request.
For AWS ES only AWSSigV4 method[1] is supported for authentication, as such if you use any other type of authorization method, the value for the "Authorization" header will not be in the correct format and ES will reject the request with the error above.
In case of fluentd the authorization header is usually added when you have the "user" and "password" parameters[2] in your elasticsearch output configuration. Since you have an open access policy on your domain, if you simply remove these two parameters from the configuration and the security groups on your domain allow connections from the clients running fluentd, you should be able to connect to your AWS ES domains without any issues.