Writetimeout is too short for index-pattern creation in Kibana about elasticsearch-proxy HOT 8 CLOSED

This may be a side effect of another issue we discovered where memory is being used because of a query across all indices to get meta data. Maybe your cluster needs more memory. Please consider posting the results of running a must gather. Instructions can be found at openshift/cluster-logging-operator

from elasticsearch-proxy.

I already increased the memory for elasticsearch-proxy as I saw those other issues but that didn't help. And even before we hadn't run in to OOM situations. As I understand from Go's http documentation, with TLS enabled the WriteTimeout is the time from request headers until response is totally written. And in our case the response from ES itself takes ~8 seconds so the elasticsearch-proxy has already closed the connection. I've tested this quite much with the Kibana's DevTools and any query taking longer than 5 seconds fails.

In any case, I think the WriteTimeout shouldn't be less than what is configured to Kibana's elasticsearch.requestTimeout (with CLO it's 300000ms). Otherwise the requests from Kibana going through elasticsearch-proxy will be disconnected after WriteTimeout.

ps. We are providing the must-gather information through a support ticket shortly.

from elasticsearch-proxy.

Could you comment on the reasoning why the elasticsearch-proxy's Go http.Server.WriteTimeout is set to 5 seconds when the requestTimeout in Kibana is set to 300 seconds?

from elasticsearch-proxy.

I have now tested this by compiling a new version of this elasticsearch-proxy where the http.Server.WriteTimeout in http.go is set to 600 seconds and instead of getting that "socket hang up" error like described in the bug description I'm able to get this (in Kibana Dev Tools tab):
{
"took" : 15495,
"timed_out" : false,
"_shards" : {
"total" : 235,
"successful" : 235,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1134384077,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"indices" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,

This means that the queries that take more than 5 seconds are successful now from Kibana to ES. Previously, anything that took longer than 5 seconds was unsuccessful as the connection was closed by elasticsearch-proxy.

from elasticsearch-proxy.

Closing fixed by #73 which bumps it to a minute. Additionally allows overriding the default configuration

from elasticsearch-proxy.

Great. I still don't understand why you would have shorter timeout at the proxy than what is configured in Kibana as the ElasticSearch queryTimeout. As I understand it, all queries from Kibana go through this elasticsearch-proxy in your implementation of ClusterLogging.

from elasticsearch-proxy.

And is the fix also coming to 4.5 or 4.6 releases? I can only see it in master and 4.7.

from elasticsearch-proxy.

Great. I still don't understand why you would have shorter timeout at the proxy than what is configured in Kibana as the ElasticSearch queryTimeout. As I understand it, all queries from Kibana go through this elasticsearch-proxy in your implementation of ClusterLogging.

The timeout was modified to address a memory issue when FIPS was enabled. I did not modify to the current value to understand what motivation was for picking that value. Regardless, I'm certain it was chosen from the perspective of the heavy write traffic from the collector and not read from Kibana. This change will be cherry-picked back to 4.5 and is awating verification from QE

from elasticsearch-proxy.