Fluentd only ships logs from it's own namespace. about fluentd HOT 16 OPEN

From @jchauncey on October 14, 2016 12:21

I am actually working on this plugin right now. It definitely was not intended for use in an environment like kubernetes. If I got you a custom image to try would you be willing to replace what you have with it? See if it works for you?

from fluentd.

From @FrederikNS on October 14, 2016 12:40

Sure, I would love to help out!

from fluentd.

From @jchauncey on October 14, 2016 12:41

k let me get a pull request setup and ill get you an image out of the deisci registry to try out. stay tuneed =)

from fluentd.

From @jchauncey on October 14, 2016 13:50

Alright so I'm not sure if this will fix your orignal problem (I never saw that problem yesterday) but this new image will allow you to do the following things:

• You can now specify how you want to index your data. Before we lumped everything into a bucket called logstash. Now you can actually use the map of data we get from fluentd to create your index name. I am using kubernetes.namespace_name in my personal cluster. (I will provide a sample configuration below).
• You can now also provide a custom index name and also use the logstash format. So you can do something like myapp_namespace-YYYY-MM-DD. This allows you to easily archive old log data.

Here is the image you can use - quay.io/deisci/fluentd:git-2aec7b0 please let me know if you have any issues. To install this image you can do 1 of hte following:

• Edit your generate_params.toml and supply the tag and org in the fluentd section. Then run helmc generate
• Edit the deis-logger-fluentd-daemon.yaml` file in the manifest directory of the chart you used to install workflow.

Then you just need to do kubectl create -f manifests/deis-logger-fluentd-daemon.yaml

        - name: "ELASTICSEARCH_LOGSTASH_FORMAT"
          value: "true"
        - name: "FLUENTD_FLUSH_INTERVAL"
          value: "10s"
        - name: TARGET_INDEX_KEY
          value: kubernetes.namespace_name

from fluentd.

From @jchauncey on October 14, 2016 16:40

Ive done a slight refactoring if you want to try this image instead - quay.io/deisci/fluentd:git-ad7196d

from fluentd.

From @FrederikNS on October 14, 2016 21:23

I just tried git-ad7196d out, and it worked beautifully. Now both the default and kube-system namespaces come through, and everything is tagged nicely. Thank you!

from fluentd.

From @jchauncey on October 14, 2016 21:39

yup. well get this merged and a release cut shortly.. waiting on some reviewers =)

from fluentd.

From @thenayr on October 18, 2016 16:25

Just chiming in on this, @jchauncey Thank you for the recent changes, this Fluentd image is much improved over the cluster default image that ships with Kubernetes.

I especially like the new TARGET_INDEX_KEY functionality!

I'm testing this out in my clusters right now using the canary image

from fluentd.

From @thenayr on October 18, 2016 20:27

After a few hours of testing with this, a few bits of feedback:

It doesn't seem possible to get the following format of index: custom-name-YYYY-MM-DD, I believe I've tried every combination of ELASTICSEARCH_LOGSTASH_FORMAT, ELASTICSEARCH_LOGSTASH_PREFIX and ELASTICSEARCH_INDEX_NAME.

Your example above with TARGET_INDEX_KEY does result in the correct index name (default-YYYY-MM-DD), but if I don't want to specify the TARGET_INDEX_KEY I always get stuck with fluentd as my index name, or whatever I specify as ELASTICSEARCH_INDEX_NAME, but with no YYYY-MM-DD suffix on the name.

Even just trying to get the default logstash-YYYY-MM-DD by setting ELASTICSEARCH_LOGSTASH_FORMAT to true doesn't want to work for me.

Also it would be nice to be able to customize reload_connections value as AWS Elasticsearch service has issues if that is set to true.

from fluentd.

From @jchauncey on October 18, 2016 20:34

k the PR is still open so i can make some small changes if you want.

You can see the logic here for writing out hte index name - https://github.com/deis/fluentd/pull/60/files#diff-5198419c579f23c85b30a0fed99ddee9R323

If you want to write all the logs to 1 index you can set ELASTICSEARCH_LOGSTASH_PREFIX while also setting ELASTICSEARCH_LOGSTASH_FORMAT

You can find these changes in - quay.io/deisci/fluentd:git-a89dfce

from fluentd.

From @thenayr on October 18, 2016 20:41

All of the feedback was from the image you just specified: quay.io/deisci/fluentd:git-a89dfce

The issue I'm having is that

- name: ELASTICSEARCH_LOGSTASH_FORMAT
  value: "true"
- name: ELASTICSEARCH_LOGSTASH_PREFIX
  value: "abc123"

Results in writing to the index name fluentd

Not sure why this is happening as the logic for index name you pointed out seems to be accounting for this correctly..

from fluentd.

From @jchauncey on October 18, 2016 20:43

That image is fairly old so I would try deploying the image I specified above which has some of the newer environment variables.

from fluentd.

From @thenayr on October 18, 2016 20:47

I'm slightly confused, the first image you referenced a few days ago was git-ad7196d, I caught that was a different commit and used the commit ID of the latest changes git-a89dfce (also the one you just specified in the previous comment).

I'm pretty certain this is the correct image git-a89dfce, I've also exec into the pod to make sure the configuration looked sound and it did.

For clarity sake, here is more of my daemonset spec:

spec:
 13       containers:
 14       - name: fluentd-logging
 15         image: quay.io/deisci/fluentd:git-a89dfce
 16         env:
 17         - name: KUBERNETES_VERIFY_SSL
 18           value: "false"
 19         - name: ELASTICSEARCH_HOST
 20           value: "elasticsearch-logging"
 21         - name: ELASTICSEARCH_LOGSTASH_FORMAT
 22           value: "true"
 23         - name: ELASTICSEARCH_LOGSTASH_PREFIX
 24           value: "abc123"

from fluentd.

From @jchauncey on October 18, 2016 20:51

Ah ok yeah if you are using git-a89dfce then you have the latest changes. =) just making sure.

Let me see if I can replicate the behavior you are seeing.

from fluentd.

From @thenayr on October 18, 2016 21:55

After some testing, this appears to be the offending line - https://github.com/deis/fluentd/pull/60/files#diff-839dc2adb331ca2e1acf44b97bdf9796R40

Removing that line I can properly set a LOGSTASH_PREFIX and have it take effect.

So it seems like even if we don't set a TARGET_INDEX_KEY it will still evaluate as "" and cause the target_index to evaluate to the default fluentd

from fluentd.

From @jchauncey on October 18, 2016 21:57

hrm k their configuration is terrible and makes it really hard to have a shell script setup those values. I will have to come up with something to help with this

from fluentd.