FileSystemWatcher in DirectorySource reliability? about kinesis-agent-windows HOT 4 CLOSED

Hi @softwareguy74 ,

Thank you very much for your suggestions. We actually do have a polling mechanism. On a busy system, the FileSystemWatcher can fire extremely rapidly. So we buffer the events and process at a timer interval. In the timer event, we do poll the file system. See:

We did observe that FileSystemEvent not alway fire, but any disturbance (such as listing files or reading the file length) seems to trigger it to fire.

So far we did not encounter this issue in our internal production. That said, we do not have many data points on UNC paths. If you encounter any problem, please do let us know.

from kinesis-agent-windows.

The issue with the UNC paths failing is when a file server is taken off line and brought back up. The FSW will fail to continue, and needs to be restarted. So there needs to be some mechanism to check if FSW is no longer working and restart it. It's possible some people put log files on UNC paths so this could be a real problem out in the real world.

from kinesis-agent-windows.

Agree. Thanks for the clarification!

We had a recent commit for a slightly difference use case but may not cover your use case: c9182fc. We'll post back on this thread once we have verified or implemented the scenario of file server taken off/brought back up.

from kinesis-agent-windows.

Hi @softwareguy74 , we have added a feature to recreate FSW when a directory becomes unavailable in bd8ba3a. Please try version 1.0.0.128 or later from our beta download page: https://s3-us-west-2.amazonaws.com/kinesis-agent-windows/beta/index.html. Let us know if the fix addresses your concern. Thank you very much.

from kinesis-agent-windows.