Describe the bug
Index-pattern creation fails in Kibana because no indices are listed with the default query

Environment

• OpenShift 4.5.15
• CLO version 4.5.0-202012120433.p0

Logs
Couldn't find much info from logs

Expected behavior
List of available indices when creating an index-pattern in Kibana

Actual behavior
When creating index-pattern in Kibana it queries the indices with this kind of POST:
URL: https://kibana-openshift-logging.apps./elasticsearch/*/_search?ignore_unavailable=true
Payload: {"size":0,"aggs":{"indices":{"terms":{"field":"_index","size":200}}}}

After a while a toast pops up saying Kibana was unable to fetch indices.

Same query using Kibana's Dev Tools gives:
{
"message": "Client request error: socket hang up",
"statusCode": 502,
"error": "Bad Gateway"
}

To Reproduce
Steps to reproduce the behavior:

1. Create an Elasticsearch cluster with enough docs.
2. Try to create index-pattern in Kibana
3. No indices are returned and can't create index-pattern

Additional context
I believe this happens because the query goes through elasticsearch-proxy and there was WriteTimeout of 5 seconds introduced in #57 . This WriteTimeout basically closes the connection if the response takes more than 5 seconds.

We have so many docs and shards because we have set the application logs retention to 30 days. Other logs (infra and audit) have retention for 7 days.

Beginning of response when same query is run from within ES pod using es_util tool tells that our query takes 8 seconds:
{
"took" : 8072,
"timed_out" : false,
"_shards" : {
"total" : 223,
"successful" : 223,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1213667064,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"indices" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "app-000050",
"doc_count" : 58109669
},
{
"key" : "app-000053",
"doc_count" : 41653740

Verify (and fix) proxy utilizes a user's token in order to fetch the projects they can see: Ref https://github.com/openshift/elasticsearch-proxy/pull/4/files#r332578304

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.6
• release-4.5

Contact the Test Platform or Automated Release teams for more information.

#4 (comment)

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.6

Contact the Test Platform or Automated Release teams for more information.

At present, ART is using the master branch for all 4.x builds.

Need integration tests to provide more assurances #4 (comment)

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.6

Contact the Test Platform or Automated Release teams for more information.

#4 (comment)

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.8

Contact the Test Platform or Automated Release teams for more information.