Randomly stops sending log to PaperTrail endpoint about fluent-plugin-papertrail HOT 9 CLOSED

Hi @cmrust ,

Thank you.

I figured out that the "I" is the common severity indetifier in Kubernetes ecosystem log messages, like in:
/var/log/kube-proxy.log

Line:
# I1118 21:26:53.975789 6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed

The FD source formatter line is:
format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/

So it should be info instead in PapertTrail output, not just "I".

In my opinion this character causes the problem. Thus I consider your second option to transform and force every event with info severity to PT.
Thanks again.

from fluent-plugin-papertrail.

Hey @tatobi,

The actual error message is error="'I' is not a designated severity". The rest of the logs are about fluentd retrying the record and incrementally backing off, probably until it fails altogether as you've noted.

This is an error from the syslog_protocol gem we use in the plugin. You're setting severity to I somewhere and it's rejecting it as non-conforming. See the following links for how it handles severity and the list of accepted values:
https://github.com/eric/syslog_protocol/blob/001000ffe27a4557c3ec312b9c3c50385e6a923b/lib/syslog_protocol/packet.rb#L61-L77
https://github.com/eric/syslog_protocol/blob/001000ffe27a4557c3ec312b9c3c50385e6a923b/lib/syslog_protocol/common.rb#L58-L78

Our plugin grabs severity from your fluent record or else defaults to 'info'. It seems you are passing the value I in to the severity field in fluent somewhere. You need to figure out where that is and change it.

See here for our plugin's implementation:
https://github.com/solarwinds/fluent-plugin-papertrail/blob/master/lib/fluent/plugin/out_papertrail.rb#L82

Let me know if that doesn't make sense.

from fluent-plugin-papertrail.

Hi @cmrust ,
Thank you for the quick response.

I use the config from here : https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml
with replace of output.conf section with PapertTrail one:

....
  output.conf: |-
    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>
    <filter kubernetes.**>
      @type record_transformer
      enable_ruby true
      <record>
        hostname kubernetes-test-${record["kubernetes"]["pod_name"]}
        program ${record["kubernetes"]["container_name"]}
        severity info
        facility local0
        message ${record['log']}
      </record>
    </filter>
    <match **>
      @type copy
      <store>
        @type papertrail
        papertrail_host logs****.papertrailapp.com
        papertrail_port *****
      </store>
    </match>

I do not set severity anywher and it is always info as you can see.
Thats it, so it is strange.

from fluent-plugin-papertrail.

In the snippet here you are transforming all records with the prefix kubernetes. and forcing severity to info.

This is good, but your match to output to papertrail is for all records **, not just ones with that prefix.

In the link you've pasted above you have many fluent sources, many of which format severity. One of these has a bad input.

I would suggest two approaches to solve your problem:

• Set an output plugin like STDOUT instead of Papertrail and watch the output of the process until you find what is logging with severity I and then go fix it.
• Create another filter with record_transformer to match all events ** and overwrite all records to have a severity of info.

Let me know how it goes @tatobi and thanks for using the plugin.

from fluent-plugin-papertrail.

The new output section looks like now, I hope it does what we want.

  output.conf: |-
    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    <filter kubernetes.**>
      @type record_transformer
      enable_ruby true
      <record>
        hostname kube-live-${record["kubernetes"]["container_name"]}
        program ${record["kubernetes"]["container_name"]}
        severity info
        facility local0
        message ${record['log']}
      </record>
    </filter>
    
    <match **>
      @type record_reformer
      enable_ruby true
      <record>
        facility local0
        message ${record['log']}
        severity info
      </record>
      
      @type copy
      <store>
        @type papertrail
        papertrail_host ***.papertrailapp.com
        papertrail_port ****
        flush_interval 60
        retry_limit 100
        retry_wait 10
      </store>
    </match>

from fluent-plugin-papertrail.

Or a second try:

  output.conf: |-
    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    <filter kubernetes.**>
      @type record_transformer
      enable_ruby true
      <record>
        hostname kube-live-${record["kubernetes"]["container_name"]}
        program ${record["kubernetes"]["container_name"]}
        severity info
        facility local0
        message ${record['log']}
      </record>
    </filter>

    <filter **>
      @type record_transformer
      enable_ruby true
      <record>
        facility local0
        message ${record['log']}
        severity info
      </record>
    </filter>
    
    <match **>
      @type copy
      <store>
        @type papertrail
        papertrail_host ***.papertrailapp.com
        papertrail_port ****
        flush_interval 60
        retry_limit 100
        retry_wait 10
      </store>
    </match>

from fluent-plugin-papertrail.

In retrospect, we could add an option to the plugin to override values like severity if they're not from the accepted lists.

from fluent-plugin-papertrail.

Thank you! Btw, after the second modification you mentioned, it looks like working well.

from fluent-plugin-papertrail.

Cheers. Glad to be of service @tatobi. Thanks for creating an issue and please don't hesitate to do so again.

from fluent-plugin-papertrail.