Comments (4)
Is it possible to share a small subset of the logs so I can take a look?
from sequence.
Here are three log lines which cause the sequencer to return the "unknown token encountered" message. The failure on the last line is understandable, given that it's a log record of a full sql statment which doens't really follow a "proper" log structure like this project expects to recieve. It would be nice if the sequencer could recover from garbage log lines, but it is certainly something I can work around with some pre-processing.
The first two lines though I was hoping to understand a little better as to why they cause the sequencer to stop.
2015-01-21 21:41:27 4515 [Note] - '::' resolves to '::';
2015-01-21 21:41:27 4515 [Note] Server socket created on IP: '::'.
2015-01-22 21:12:14 4515 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. INSERT... ON DUPLICATE KEY UPDATE on a table with more than one UNIQUE KEY is unsafe Statement: INSERT INTO surf_swell_filter_predicates ( surf_id, swell_id, element_id, predicate_type, action, has_extra_data,device_id,plugin_id,object_id,indicator_id,object_type_id,indicator_type_id,device_class_id,device_group_id,object_group_id ) VALUES ( 13,49,206,"flowInterface","=","1",NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL ) ON DUPLICATE KEY UPDATE predicate_type = VALUES(predicate_type), action = VALUES(action), has_extra_data = VALUES(has_extra_data), device_id = VALUES(device_id), plugin_id = VALUES(plugin_id), object_id = VALUES(object_id), indicator_id = VALUES(indicator_id), object_type_id = VALUES(object_type_id), indicator_type_id = VALUES(indicator_type_id), device_class_id = VALUES(device_class_id), device_group_id = VALUES(device_group_id), object_group_id = VALUES(object_group_id)
from sequence.
Thanks @wkrause13 . You hit a bug where I was not recognizing "::" as a valid IPv6 address correctly (actually I think I was lazy and completely missed it.)
I just checked in a fix and the first two logs seem to be tokenizing correctly now. Can you let me know if it works for you?
thx
$ go run ./sequence.go scan -m "2015-01-21 21:41:27 4515 [Note] - '::' resolves to '::';"
# 0: { Field="%funknown%", Type="%time%", Value="2015-01-21 21:41:27", K=false, V=false }
# 1: { Field="%funknown%", Type="%integer%", Value="4515", K=false, V=false }
# 2: { Field="%funknown%", Type="%literal%", Value="[", K=false, V=false }
# 3: { Field="%funknown%", Type="%literal%", Value="Note", K=false, V=false }
# 4: { Field="%funknown%", Type="%literal%", Value="]", K=false, V=false }
# 5: { Field="%funknown%", Type="%literal%", Value="-", K=false, V=false }
# 6: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 7: { Field="%funknown%", Type="%ipv6%", Value="::", K=false, V=false }
# 8: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 9: { Field="%funknown%", Type="%literal%", Value="resolves", K=false, V=false }
# 10: { Field="%funknown%", Type="%literal%", Value="to", K=false, V=false }
# 11: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 12: { Field="%funknown%", Type="%ipv6%", Value="::", K=false, V=false }
# 13: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 14: { Field="%funknown%", Type="%literal%", Value=";", K=false, V=false }
$ go run ./sequence.go scan -m "2015-01-21 21:41:27 4515 [Note] Server socket created on IP: ':
:'."
# 0: { Field="%funknown%", Type="%time%", Value="2015-01-21 21:41:27", K=false, V=false }
# 1: { Field="%funknown%", Type="%integer%", Value="4515", K=false, V=false }
# 2: { Field="%funknown%", Type="%literal%", Value="[", K=false, V=false }
# 3: { Field="%funknown%", Type="%literal%", Value="Note", K=false, V=false }
# 4: { Field="%funknown%", Type="%literal%", Value="]", K=false, V=false }
# 5: { Field="%funknown%", Type="%literal%", Value="Server", K=false, V=false }
# 6: { Field="%funknown%", Type="%literal%", Value="socket", K=false, V=false }
# 7: { Field="%funknown%", Type="%literal%", Value="created", K=false, V=false }
# 8: { Field="%funknown%", Type="%literal%", Value="on", K=false, V=false }
# 9: { Field="%funknown%", Type="%literal%", Value="IP", K=false, V=false }
# 10: { Field="%funknown%", Type="%literal%", Value=":", K=false, V=false }
# 11: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 12: { Field="%funknown%", Type="%ipv6%", Value="::", K=false, V=false }
# 13: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 14: { Field="%funknown%", Type="%literal%", Value=".", K=false, V=false }
from sequence.
@wkrause13 assume things are working so closing this...let me know if you have any questions..thx
from sequence.
Related Issues (20)
- Readme doesn't say how to run / build HOT 6
- Sequence can't handle Chinese HOT 2
- No output to `stdout` when output file flag is empty HOT 1
- Sequences with URI's are not matched correctly HOT 1
- URI's starting with "//" are not tokenized correctly HOT 1
- '|' (pipe character) causes error during analyze HOT 2
- Path not correctly analyse HOT 1
- ipv6 address not tokenized properly HOT 2
- syslog-ng patterndb integration HOT 1
- data folder is missing in the root directory HOT 1
- Been working on sequence for the last 2 months - would love to discuss HOT 3
- Why does the html tag identify as a tag. HOT 1
- unit test failed in analyzer_test.go HOT 1
- integer with trailing dot recognized as float HOT 1
- time not recognized at end of line HOT 2
- A greedy (whitespace-consuming) %string% type? HOT 6
- relaxing types HOT 3
- show message counts HOT 3
- iostat output - split over multiple lines, and in multi-line table HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sequence.