read_from_head activates if you close fluentd too soon about fluent-plugin-windows-eventlog HOT 10 CLOSED

After some more digging I found that some of the channels in the storage file looks like this

"hardwareevents": "<BookmarkList Direction='backward'>\r\n</BookmarkList>",
So for some reason some channels gets Direction='backward', even while running for the first time, but it will dissapear if a new log comes in, and the backward direction will only be triggered if you restart.

from fluent-plugin-windows-eventlog.

I'm developing this issue's workaround.
With #42 patch, we can subscribe channels like as:

<subscribe>
  channels ["TargetChannel"]
  read_existing_events true
</subscribe>
<subscribe>
  channels ["TargetChannel"]
  read_existing_events false
</subscribe>

This will be handled as the following when without bookmark:

• The prior case is subscribing TargetChannel with EvtSubscribeToFutureEvents
• The latter case is subscribing TargetChannel with EvtSubscribeStartAtOldestRecord

ref: https://docs.microsoft.com/en-us/windows/win32/api/winevt/ne-winevt-evt_subscribe_flags

But I have no idea why the empty channel is not consumed until Fluentd restarts....

from fluent-plugin-windows-eventlog.

I'd created the patch to validate bookmark contents.
Could you try it out, @flurreN ?

Works for me!

from fluent-plugin-windows-eventlog.

Thanks for the detailed report.
I also got Direction='backward' result when subscribing empty channel.
I'm considering how to fix it or find workaround.

from fluent-plugin-windows-eventlog.

Not sure if read_existing_events was suppose to fix it but after some test runs I got this.

channels Security,Application
read_existing_events false
Run 1: {"application":"<BookmarkList Direction='backward'>\r\n</BookmarkList>","security":"<BookmarkList>\r\n  <Bookmark Channel='Security' RecordId='23460' IsCurrent='true'/>\r\n</BookmarkList>"}
Run 2: {"application":"<BookmarkList>\r\n  <Bookmark Channel='Application' RecordId='931' IsCurrent='true'/>\r\n</BookmarkList>","security":"<BookmarkList>\r\n  <Bookmark Channel='Security' RecordId='23460' IsCurrent='true'/>\r\n</BookmarkList>"}

channels Security,Application
read_existing_events false
Run 1: {"application":"<BookmarkList>\r\n</BookmarkList>","security":"<BookmarkList>\r\n  <Bookmark Channel='Security' RecordId='23464' IsCurrent='true'/>\r\n</BookmarkList>"}
Run 2: {"application":"<BookmarkList>\r\n  <Bookmark Channel='Application' RecordId='600' IsCurrent='true'/>\r\n</BookmarkList>","security":"<BookmarkList>\r\n  <Bookmark Channel='Security' RecordId='23464' IsCurrent='true'/>\r\n</BookmarkList>"}

channels application
read_existing_events false
Run 1: {"application":"<BookmarkList Direction='backward'>\r\n</BookmarkList>"}
Run 2: {"application":"<BookmarkList>\r\n  <Bookmark Channel='Application' RecordId='201' IsCurrent='true'/>\r\n</BookmarkList>"}

Same results with:

<subscribe>
  channels ["Application"]
  read_existing_events false
</subscribe>

Isnt it possible to make something like <BookmarkList Direction='forward'>? and force it to wait until a new log comes is, because if not closing fluentd to soon and for example a log comes into "Application" then it will continue from that point.

from fluent-plugin-windows-eventlog.

Isnt it possible to make something like ?

No. This behavior is from Windows API which is provided by winevt.h. We cannot customize them.

from fluent-plugin-windows-eventlog.

Looks like because fluentd always write out {"application":"<BookmarkList'>\r\n</BookmarkList>"} it will always trigger the first if statement in this code on the second fluentd run, because the bookmark already exist on the 2nd run.
Because it says EvtSubscribeStartAfterBookmark and the bookmark is incomplete , it will start from the begining. Looks like thats the way EvtSubscribeStartAfterBookmark is working.

  if (hBookmark) {
    flags |= EvtSubscribeStartAfterBookmark;
  } else if (winevtSubscribe->readExistingEvents) {
    flags |= EvtSubscribeStartAtOldestRecord;
  } else {
    flags |= EvtSubscribeToFutureEvents;
  }

So to fix this is it possible to rewrite this and check if the bookmark actually is complete?

But at the same time an uncomplete bookmark like {"application":"<BookmarkList'>\r\n</BookmarkList>"} cant be used with EvtSubscribeToFutureEvents

from fluent-plugin-windows-eventlog.

So to fix this is it possible to rewrite this and check if the bookmark actually is complete?

In winevt_c extension side, it is not possible.
But in Ruby side with nokogiri, we can check incomplete bookmark with XML parsing.

Because it says EvtSubscribeStartAfterBookmark and the bookmark is incomplete , it will start from the begining. Looks like thats the way EvtSubscribeStartAfterBookmark is working.

Yep. You are right. I'm thinking of how to handle incomplete bookmark mechanism.

from fluent-plugin-windows-eventlog.

Isnt it possible to make something like <BookmarkList Direction='forward'>?

Without Direction means Direction='forward'.

require 'winevt'

bookmark_str = <<EOS
<BookmarkList Direction='forward'>
</BookmarkList>
EOS

bookmark = Winevt::EventLog::Bookmark.new(bookmark_str)
bookmark.render #=> "<BookmarkList>\r\n</BookmarkList>"

from fluent-plugin-windows-eventlog.

I'd created the patch to validate bookmark contents.
Could you try it out, @flurreN ?

from fluent-plugin-windows-eventlog.