zero-sized unified2 files causes problem about py-idstools HOT 9 CLOSED

It should read newer files.. Does it appear to be stuck on this file and not progressing to newer ones? What are the timestamps of the other files?

from py-idstools.

It does seem to get stuck, until I remove the bookmark and zero-sized files. The timestamps on the other files are afterwards ... i.e. the order is chronological as they happened via snort.

from py-idstools.

Have a test (manual, automated, whatever) that can demonstrate this? I just tested with idstools-u2json which uses unified2.SpoolEventReader. I created a 0 size unified2 file in my log directory with a timestamp newer than the existing one. I then restarted Snort which results in an even newer one being created. unified2.SpoolEventReader opened the 0 size file, deleted it then continued onto the newer file. What arguments are you creating SpoolEventReader with?

0 size files aren't that uncommon. If Snort is started, then restarted before any events are logged, you will get 0 sized unified2 files.

from py-idstools.

Well, I will have to check with the client for the files that currently cause this, but it sounds like you've tried the same situation but with success.
My call to SpoolEventReader is this:

    u2json_reader = unified2.SpoolEventReader(
      directory=args.unified2_directory,
      prefix=args.unified2_prefix,
      follow=args.unified2_follow,
      bookmark=args.unified2_bookmark
    )

... which is used like so:

  for ok, result in streaming_bulk(
      client,
      read_events_as_json(u2json_reader),
      chunk_size=int(chunk_size)
    ):

... where read_events_as_json is a generator like:

    for event in u2json_reader:
      test_serializer = JSONSerializer().dumps(encoded_json)
      yield encoded_json

... if that helps.

from py-idstools.

I didn't catch this in your response ...
"delete – If True, unified2 files will be deleted when reading has moved onto the next one."
... is that a required setting for this to work ?
I just assumed the bookmark was "remembering" where it was and in which file, thus skipping
any older files.

from py-idstools.

Just tested without delete=True. Still works. Have you configured logging? With a logger configured, the SpoolEventReader will log to stderr lines like:

Closed file /var/log/snort/unified2.log.1413323322, opened file /var/log/snort/unified2.log.1413401177
Closed file /var/log/snort/unified2.log.1413401177, opened file /var/log/snort/unified2.log.1413401186
Closed file /var/log/snort/unified2.log.1413401186, opened file /var/log/snort/unified2.log.1413401263

In this case I was running u2json like:

python /usr/bin/idstools-u2json --snort-conf=etc/snort.conf
--directory=/var/log/snort
--prefix=unified2.log
--output=/var/log/snort/alerts.json
--follow
--bookmark
--stdout
--packets

The first 2 log lines are the result of me doing "touch unified2.log.$(date +%s)" in order to generate 0 sized files. The third is the result of me restarting Snort. For me its working as designed which is:

• Read from currently opened file.
• If no data read, check directory for a newer file.
• If newer file exists, open it, and optionally delete the previous open file (delete=True)

from py-idstools.

If I may ask, how would I configure a logger ? I have this:

import logging
logging.basicConfig(level=logging.INFO, format="%(message)s")
LOG = logging.getLogger("U2ELog")

... but I don't see messages like "Closed file ...".
As I'm not using the idstools' scripts in bin, perhaps I've missed something.

Also, our upstart script is:

description "U2E Indexer"
start on runlevel [2345]
stop on runlevel [016]
chdir /some/folder/u2e_indexer
exec /usr/bin/python u2e_indexer.py @u2e.args >> /var/log/u2e_indexer/indexer.log 2>&1

... I thought the ">> /var/log/u2e_indexer/indexer.log 2>&1" would redirect all stdout/stderr to
the log file.
Again, thanks for all of the info/testing.

from py-idstools.

That should do it.. For example, the u2json entry script has:

import logging
logging.basicConfig(level=logging.INFO, format="%(message)s")

And the unified2.SpoolEventReader will call:

        if closed:
            LOG.info("Closed file %s, opened file %s", closed, opened)
        else:
            LOG.info("Opened file %s", opened)

So yes, it looks like it should end up in your indexer.log.

Sorry I can't be much more help - until I can reproduce the issue of course.

from py-idstools.

I'm sure there's something in my code or our setup, so I'll close this but thanks for all your help.

from py-idstools.