cyberdelia / logfmt-ruby Goto Github PK

I discovered this in a logger I was writing inspired by this implementation.

The problem is that this particular ActionDispatch logging looks specifically for tags_text which is fine for single-value tags, but not for hash key/value tags.

https://github.com/rails/rails/blob/fc734f28e65ef8829a1a939ee6702c1f349a1d5a/actionpack/lib/action_dispatch/middleware/debug_exceptions.rb#L155

My own solution was to undef :tags_text in the formatter.

We have logfmt logs in the below format.

=>Logfmt.parse{"foo"=>"bar", "a"=>14, "baz"=>"hello\" kitty"}
=> {"foo"=>"bar", "a"=>14, "baz"=>"hello", kitty"=true}

Can we use the escape character while parsing the logs?
Can we drop such values or the empty one?

I've found that the last value will always get set to "true" if the value is only 1 character long. For instance:

'key1=4 key2=9'

ends up generating the value

{"key1"=>"4", "key2"=>true}

Here's a test case that will demonstrate the issue:

  it 'parse last as integer type' do
    data = Logfmt.parse('key1=4 key2=9')
    expect(data["key1"]).to eq(4)
    expect(data["key2"]).to eq(9)
  end

The error that gets generated is:

  1) Logfmt parse last as integer type
     Failure/Error: expect(data["key2"]).to eq(9)

       expected: 9
            got: true

       (compared using ==)

I haven't quite pinpointed where the bug is. It seems like these lines initially set the value to true, but the proper setting of the value never happens when the value is of length 1.

Thanks

I understand this is not an issue and but I am trying to understand the reason behind it.

>> require "logfmt"
>> Logfmt.parse('foo=bar a=14 baz="hello kitty" cool%story=bro f %^asdf')
=> {"foo"=>"bar", "a"=>14, "baz"=>"hello kitty", "cool%story"=>"bro", "f"=>true, "%^asdf"=>true}`

Why null values have to get converted as true
Parsing the quotes in the string have similar behaviors. i.e "something"test"

In the documentation from http://godoc.org/github.com/kr/logfmt, the parsed JSON ends up being:

{ "foo": "bar", "a": 14, "baz": "hello kitty", "cool%story": "bro", "f": true, "%^asdf": true }

Both the Ruby & Python versions end up generating the following JSON:

{"foo":"bar", "a":"14", "baz":"hello kitty", "cool%story":"bro", "f":true, "%^asdf":true}

Even though "a" was originally an integer, it is now a string. This ends up becoming an issue when log data is sent into a data store like ElasticSearch which preserves the JSON types posted to it. If you try running aggregation queries such as min, max, and avg in ElasticSearch on a field that is of type string, you end up getting a ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData error.
This issue sort of breaks the flow for logging in key value pairs, parsing that data, & sending that data to data store that properly supports JSON types.

Any ideas on how this can be fixed? I noticed in the original logfmt project, a formatter was implemented that would make sure to generate the correct JSON type. I'm guessing that somewhere in here there should be an isFloat/isInteger test and value type conversion.

Thanks

Given this line as input:

level=event ts=2018-10-29T14:52:51.473589976Z msg="flush chunks" userID=11463 reason=2 numChunks=1 firstTime=1540821644.498 fp=524157e785101116 series="node_cpu{_weave_service=\"prom-node-exporter\",cpu=\"cpu5\",instance=\"10.132.0.4:9100\",job=\"kubernetes-pods\",kubernetes_namespace=\"weave\",kubernetes_pod_name=\"prom-node-exporter-c97bk\",mode=\"system\",node=\"gke-cluster-1\"}" queue=8

fp comes out as Infinity. I suspect this is because it matches the float regexp, but when parsed as a float it is too big.

Is it possible to have the logfmt parser read this field as a string?

Hello,

I noticed an issue when using this gem to parse my HTTP request logs. I have a path that has an equals sign in it and the parser is breaking my path into two key-value pairs, which is not correct. To reproduce run the following:

Logfmt.parse("path=/mykey=/morepath format=html")

The result is:
{"path"=>"/mykey", "/morepath"=>true, "format"=>"html"}

However, the result should be:
{"path"=>"/mykey/morepath", "format"=>"html"}

Cheers,