cyberdelia / logfmt-ruby Goto Github PK
View Code? Open in Web Editor NEWParse log lines in the logfmt style.
Home Page: https://github.com/cyberdelia/logfmt-ruby
Parse log lines in the logfmt style.
Home Page: https://github.com/cyberdelia/logfmt-ruby
I discovered this in a logger I was writing inspired by this implementation.
The problem is that this particular ActionDispatch logging looks specifically for tags_text
which is fine for single-value tags, but not for hash key/value tags.
My own solution was to undef :tags_text
in the formatter.
We have logfmt logs in the below format.
=>Logfmt.parse{"foo"=>"bar", "a"=>14, "baz"=>"hello\" kitty"}
=> {"foo"=>"bar", "a"=>14, "baz"=>"hello", kitty"=true}
Can we use the escape character while parsing the logs?
Can we drop such values or the empty one?
I've found that the last value will always get set to "true" if the value is only 1 character long. For instance:
'key1=4 key2=9'
ends up generating the value
{"key1"=>"4", "key2"=>true}
Here's a test case that will demonstrate the issue:
it 'parse last as integer type' do
data = Logfmt.parse('key1=4 key2=9')
expect(data["key1"]).to eq(4)
expect(data["key2"]).to eq(9)
end
The error that gets generated is:
1) Logfmt parse last as integer type
Failure/Error: expect(data["key2"]).to eq(9)
expected: 9
got: true
(compared using ==)
I haven't quite pinpointed where the bug is. It seems like these lines initially set the value to true, but the proper setting of the value never happens when the value is of length 1.
Thanks
I understand this is not an issue and but I am trying to understand the reason behind it.
>> require "logfmt"
>> Logfmt.parse('foo=bar a=14 baz="hello kitty" cool%story=bro f %^asdf')
=> {"foo"=>"bar", "a"=>14, "baz"=>"hello kitty", "cool%story"=>"bro", "f"=>true, "%^asdf"=>true}`
Why null values have to get converted as true
Parsing the quotes in the string have similar behaviors. i.e "something"test"
In the documentation from http://godoc.org/github.com/kr/logfmt, the parsed JSON ends up being:
{ "foo": "bar", "a": 14, "baz": "hello kitty", "cool%story": "bro", "f": true, "%^asdf": true }
Both the Ruby & Python versions end up generating the following JSON:
{"foo":"bar", "a":"14", "baz":"hello kitty", "cool%story":"bro", "f":true, "%^asdf":true}
Even though "a" was originally an integer, it is now a string. This ends up becoming an issue when log data is sent into a data store like ElasticSearch which preserves the JSON types posted to it. If you try running aggregation queries such as min, max, and avg in ElasticSearch on a field that is of type string, you end up getting a ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData
error.
This issue sort of breaks the flow for logging in key value pairs, parsing that data, & sending that data to data store that properly supports JSON types.
Any ideas on how this can be fixed? I noticed in the original logfmt project, a formatter was implemented that would make sure to generate the correct JSON type. I'm guessing that somewhere in here there should be an isFloat/isInteger test and value type conversion.
Thanks
Given this line as input:
level=event ts=2018-10-29T14:52:51.473589976Z msg="flush chunks" userID=11463 reason=2 numChunks=1 firstTime=1540821644.498 fp=524157e785101116 series="node_cpu{_weave_service=\"prom-node-exporter\",cpu=\"cpu5\",instance=\"10.132.0.4:9100\",job=\"kubernetes-pods\",kubernetes_namespace=\"weave\",kubernetes_pod_name=\"prom-node-exporter-c97bk\",mode=\"system\",node=\"gke-cluster-1\"}" queue=8
fp
comes out as Infinity
. I suspect this is because it matches the float regexp, but when parsed as a float it is too big.
Is it possible to have the logfmt parser read this field as a string?
Hello,
I noticed an issue when using this gem to parse my HTTP request logs. I have a path that has an equals sign in it and the parser is breaking my path into two key-value pairs, which is not correct. To reproduce run the following:
Logfmt.parse("path=/mykey=/morepath format=html")
The result is:
{"path"=>"/mykey", "/morepath"=>true, "format"=>"html"}
However, the result should be:
{"path"=>"/mykey/morepath", "format"=>"html"}
Cheers,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.