Comments (4)
zhenjl
commented on August 19, 2024
Is it possible to share a small subset of the logs so I can take a look?
from sequence.
wkrause13
commented on August 19, 2024
Here are three log lines which cause the sequencer to return the "unknown token encountered" message. The failure on the last line is understandable, given that it's a log record of a full sql statment which doens't really follow a "proper" log structure like this project expects to recieve. It would be nice if the sequencer could recover from garbage log lines, but it is certainly something I can work around with some pre-processing.
The first two lines though I was hoping to understand a little better as to why they cause the sequencer to stop.
2015-01-21 21:41:27 4515 [Note] - '::' resolves to '::';
2015-01-21 21:41:27 4515 [Note] Server socket created on IP: '::'.
2015-01-22 21:12:14 4515 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. INSERT... ON DUPLICATE KEY UPDATE on a table with more than one UNIQUE KEY is unsafe Statement: INSERT INTO surf_swell_filter_predicates ( surf_id, swell_id, element_id, predicate_type, action, has_extra_data,device_id,plugin_id,object_id,indicator_id,object_type_id,indicator_type_id,device_class_id,device_group_id,object_group_id ) VALUES ( 13,49,206,"flowInterface","=","1",NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL ) ON DUPLICATE KEY UPDATE predicate_type = VALUES(predicate_type), action = VALUES(action), has_extra_data = VALUES(has_extra_data), device_id = VALUES(device_id), plugin_id = VALUES(plugin_id), object_id = VALUES(object_id), indicator_id = VALUES(indicator_id), object_type_id = VALUES(object_type_id), indicator_type_id = VALUES(indicator_type_id), device_class_id = VALUES(device_class_id), device_group_id = VALUES(device_group_id), object_group_id = VALUES(object_group_id)
from sequence.
zhenjl
commented on August 19, 2024
Thanks @wkrause13 . You hit a bug where I was not recognizing "::" as a valid IPv6 address correctly (actually I think I was lazy and completely missed it.)
I just checked in a fix and the first two logs seem to be tokenizing correctly now. Can you let me know if it works for you?
thx
$ go run ./sequence.go scan -m "2015-01-21 21:41:27 4515 [Note] - '::' resolves to '::';"
# 0: { Field="%funknown%", Type="%time%", Value="2015-01-21 21:41:27", K=false, V=false }
# 1: { Field="%funknown%", Type="%integer%", Value="4515", K=false, V=false }
# 2: { Field="%funknown%", Type="%literal%", Value="[", K=false, V=false }
# 3: { Field="%funknown%", Type="%literal%", Value="Note", K=false, V=false }
# 4: { Field="%funknown%", Type="%literal%", Value="]", K=false, V=false }
# 5: { Field="%funknown%", Type="%literal%", Value="-", K=false, V=false }
# 6: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 7: { Field="%funknown%", Type="%ipv6%", Value="::", K=false, V=false }
# 8: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 9: { Field="%funknown%", Type="%literal%", Value="resolves", K=false, V=false }
# 10: { Field="%funknown%", Type="%literal%", Value="to", K=false, V=false }
# 11: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 12: { Field="%funknown%", Type="%ipv6%", Value="::", K=false, V=false }
# 13: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 14: { Field="%funknown%", Type="%literal%", Value=";", K=false, V=false }
$ go run ./sequence.go scan -m "2015-01-21 21:41:27 4515 [Note] Server socket created on IP: ':
:'."
# 0: { Field="%funknown%", Type="%time%", Value="2015-01-21 21:41:27", K=false, V=false }
# 1: { Field="%funknown%", Type="%integer%", Value="4515", K=false, V=false }
# 2: { Field="%funknown%", Type="%literal%", Value="[", K=false, V=false }
# 3: { Field="%funknown%", Type="%literal%", Value="Note", K=false, V=false }
# 4: { Field="%funknown%", Type="%literal%", Value="]", K=false, V=false }
# 5: { Field="%funknown%", Type="%literal%", Value="Server", K=false, V=false }
# 6: { Field="%funknown%", Type="%literal%", Value="socket", K=false, V=false }
# 7: { Field="%funknown%", Type="%literal%", Value="created", K=false, V=false }
# 8: { Field="%funknown%", Type="%literal%", Value="on", K=false, V=false }
# 9: { Field="%funknown%", Type="%literal%", Value="IP", K=false, V=false }
# 10: { Field="%funknown%", Type="%literal%", Value=":", K=false, V=false }
# 11: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 12: { Field="%funknown%", Type="%ipv6%", Value="::", K=false, V=false }
# 13: { Field="%funknown%", Type="%literal%", Value="'", K=false, V=false }
# 14: { Field="%funknown%", Type="%literal%", Value=".", K=false, V=false }
from sequence.
zhenjl
commented on August 19, 2024
@wkrause13 assume things are working so closing this...let me know if you have any questions..thx
from sequence.
Related Issues (20)
- Readme doesn't say how to run / build HOT 6
- Sequence can't handle Chinese HOT 2
- No output to `stdout` when output file flag is empty HOT 1
- Sequences with URI's are not matched correctly HOT 1
- URI's starting with "//" are not tokenized correctly HOT 1
- '|' (pipe character) causes error during analyze HOT 2
- Path not correctly analyse HOT 1
- ipv6 address not tokenized properly HOT 2
- syslog-ng patterndb integration HOT 1
- data folder is missing in the root directory HOT 1
- Been working on sequence for the last 2 months - would love to discuss HOT 3
- Why does the html tag identify as a tag. HOT 1
- unit test failed in analyzer_test.go HOT 1
- integer with trailing dot recognized as float HOT 1
- time not recognized at end of line HOT 2
- A greedy (whitespace-consuming) %string% type? HOT 6
- relaxing types HOT 3
- show message counts HOT 3
- iostat output - split over multiple lines, and in multi-line table HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sequence.