Encoding issue crash about py-idstools HOT 7 CLOSED

If using master from git, pull in commit 67a0dbb which will log an error message with the failing record to the error stream, the continue processing the log.

I thought I had handled all encoding issues. Any chance you can share a failing log file?

from py-idstools.

Ah ok looks like master has the fix that i'm looking for. Thank you for
directing me to that!

On Tue, Jun 30, 2015 at 6:33 PM, Jason Ish [email protected] wrote:

If using master from git, pull in commit 67a0dbb
67a0dbb
which will log an error message with the failing record to the error
stream, the continue processing the log.

I thought I had handled all encoding issues. Any chance you can share a
failing log file?

—
Reply to this email directly or view it on GitHub
#19 (comment)
.

from py-idstools.

Looks like its still failing, just not crashing. Here is a packet that has
issues:

ERROR: Failed to encode record as JSON: 'utf8' codec can't decode byte 0xec
in position 633: invalid continuation byte: {'data': '"
type="application/rss+xml" title="Job Blog » Moon Power is Getting
Hotter Comments Feed" href="
https://www.test.com/blog/2016/04/this-name-is-getting-weird.html/feed"
/>\n\t\t<script type="text/javascript"> window._wpemojiSettings =
{"baseUrl":"http://s.w.org
/images/core/emoji/72x72/","ext":".png","source":{"concatemoji":"https://
www.test.com\\/blog\\/wp-includes\\/js\\/wp-name-release.min.js?ver=4.2.2"}};
!function(a,b,c){function d(a){var
c=b.createElement("canvas"),d=c.getContext&&c.getContext("2d");return
d&&d.fillText?(d.textBaseline="top",d.font="600 32px
Arial","flag"===a?(d.fillText(<\xec<\xe7,0,0),c.toDataURL().length>3e3):(d.fillText(=\x03,0,0),0!==d.getImageData(16,16,1,1).data[0])):!1}function
e(a){var
c=b.createElement("script");c.src=a,c.type="text/javascript",b.getElementsByTagName("head")[0].appendChild(c)}var
f,g;c.supports={simple:d("simple"),flag:d("flag")},c.DOMReady=!1,c.readyCallback=function(){c.DOMReady=!0},c.supports.simple&&c.supports.flag||(g=function(){c.readyCallback()},b.addEventListener?(b.addEventListener("DOMContentLoaded",g,!1),a.addEventListener("load",g,!1)):(a.attachEvent("onload",g),b.attachEvent("onreadystatechange",function(){"complete"===b.readyState&&c.readyCallbac
', 'sensor-id': 0, 'event-second': 1435710674, 'data-type': 1,
'data-length': 1259, 'event-type': 4, 'event-length': 1283, 'type': 13,
'event-id': 21022}

On Wed, Jul 1, 2015 at 10:09 AM, Daniel Kasen [email protected] wrote:

Ah ok looks like master has the fix that i'm looking for. Thank you for
directing me to that!

On Tue, Jun 30, 2015 at 6:33 PM, Jason Ish [email protected]
wrote:

If using master from git, pull in commit 67a0dbb
67a0dbb
which will log an error message with the failing record to the error
stream, the continue processing the log.

I thought I had handled all encoding issues. Any chance you can share a
failing log file?

—
Reply to this email directly or view it on GitHub
#19 (comment)
.

from py-idstools.

Hello Jason,
I was also wondering if there is a way to merge the event with the packet
data into 1 JSON blob. I enjoy the modularity offered at the moment, but it
requires some interesting hacks to get them to show up together as an
elasticsearch event.

Thanks,
Daniel Kasen

On Wed, Jul 1, 2015 at 10:22 AM, Daniel Kasen [email protected] wrote:

Looks like its still failing, just not crashing. Here is a packet that has
issues:

ERROR: Failed to encode record as JSON: 'utf8' codec can't decode byte
0xec in position 633: invalid continuation byte: {'data': '"
type="application/rss+xml" title="Job Blog » Moon Power is Getting
Hotter Comments Feed" href="
https://www.test.com/blog/2016/04/this-name-is-getting-weird.html/feed"
/>\n\t\t<script type="text/javascript"> window._wpemojiSettings =
{"baseUrl":"http://s.w.org
/images/core/emoji/72x72/","ext":".png","source":{"concatemoji":"https://
www.test.com\\/blog\\/wp-includes\\/js\\/wp-name-release.min.js?ver=4.2.2"}};
!function(a,b,c){function d(a){var
c=b.createElement("canvas"),d=c.getContext&&c.getContext("2d");return
d&&d.fillText?(d.textBaseline="top",d.font="600 32px
Arial","flag"===a?(d.fillText(<\xec<\xe7,0,0),c.toDataURL().length>3e3):(d.fillText(=\x03,0,0),0!==d.getImageData(16,16,1,1).data[0])):!1}function
e(a){var
c=b.createElement("script");c.src=a,c.type="text/javascript",b.getElementsByTagName("head")[0].appendChild(c)}var
f,g;c.supports={simple:d("simple"),flag:d("flag")},c.DOMReady=!1,c.readyCallback=function(){c.DOMReady=!0},c.supports.simple&&c.supports.flag||(g=function(){c.readyCallback()},b.addEventListener?(b.addEventListener("DOMContentLoaded",g,!1),a.addEventListener("load",g,!1)):(a.attachEvent("onload",g),b.attachEvent("onreadystatechange",function(){"complete"===b.readyState&&c.readyCallbac
', 'sensor-id': 0, 'event-second': 1435710674, 'data-type': 1,
'data-length': 1259, 'event-type': 4, 'event-length': 1283, 'type': 13,
'event-id': 21022}

On Wed, Jul 1, 2015 at 10:09 AM, Daniel Kasen [email protected] wrote:

Ah ok looks like master has the fix that i'm looking for. Thank you for
directing me to that!

On Tue, Jun 30, 2015 at 6:33 PM, Jason Ish [email protected]
wrote:

If using master from git, pull in commit 67a0dbb
67a0dbb
which will log an error message with the failing record to the error
stream, the continue processing the log.

I thought I had handled all encoding issues. Any chance you can share a
failing log file?

—
Reply to this email directly or view it on GitHub
#19 (comment)
.

from py-idstools.

I was also wondering if there is a way to merge the event with the packet
data into 1 JSON blob. I enjoy the modularity offered at the moment, but it
requires some interesting hacks to get them to show up together as an
elasticsearch event.

Yes, check out idstools-u2eve. It outputs a JSON blob that combines the event record and the first associated packet which may be useful to you.

But otherwise, no I don't have a way to merge the event, all the packets and extra data that may be associated with an event. This is because the order of the records in the file may not be correct, at least for packets. For example you may have an event record, packet record, another event record, and packet record, then a packet record for the previous event record.

So to avoid the buffering, and protential delay of getting the event out, I didn't want to deal with it in a generic way.

I'll look at the other issues in the next few days likely.

from py-idstools.

@djtecha I've changed the way JSON is encoded, hopefully it resolves this issue for you. Its in master.

from py-idstools.

It worked wonderfully thank you.

from py-idstools.