Could not get the filesystem event about efsw HOT 13 CLOSED

Original comment by Martín Lucas Golini (Bitbucket: SpartanJ, GitHub: SpartanJ).

Hi!

You're doing it wrong. fileWatcher->watch() should be called only once, it's asynchronous. Please refer to the test file to understand how it should be done.

Regards,
Martín

from efsw.

Original comment by Sarvagya Pant (Bitbucket: sarvpant, ).

I have modified the code so that watch is called only once. But the problem still persist. There is a logger that writes in output.log file. When the code was executed, efsw could detect change in output.log file, but as the time passed, even though contents are being updated, efsw could not detect change. If the output.log file is opened, efsw detects. Is this a bug or am I missing something?

from efsw.

Original comment by Sarvagya Pant (Bitbucket: sarvpant, ).

Code is updated but the issue still persist.

from efsw.

Original comment by Martín Lucas Golini (Bitbucket: SpartanJ, GitHub: SpartanJ).

The library reports a "Modified" event when the file descriptor is closed, it doesn't report when the file last write date changes, this can be changed but it's not supported by the library ( i can tell you how ). This is the second time that someone ask for this feature, so i'm open to add this as an optional feature. Can you tell me your use case? Because the last time i couldn't find a reason to add this.

from efsw.

Original comment by Sarvagya Pant (Bitbucket: sarvpant, ).

I am using efsw as filesystem monitoring. When some log file gets modified, I have to read its content and send the content to server. But since the logger writes the log file, even though it gets contents written, efsw could not trigger Modified event. Thus, even the contents are being written in log file, my program could not send the logs to the server. If I were to fix this, how should I proceed? Does my code require some tweak or should i change in efsw. Thank you.

from efsw.

Original comment by Martín Lucas Golini (Bitbucket: SpartanJ, GitHub: SpartanJ).

Hi!
The solution depends in if you have control of the log files or not ( and also what you prefer to do ):

In the case you have control over the writed log files, you just need to close the file descriptor after writing the data, so you open the file, write, and close when you wan't to notify the event.

In the other case, depends on the OS you are using this, for Linux ( INotify ) you can see the solution in this issue, that is just adding the flag IN_MODIFY in the FileWatcherInotify.cpp file, line 128, from:

#!c++

int wd = inotify_add_watch (mFD, dir.c_str(), IN_CLOSE_WRITE | IN_MOVED_TO | IN_CREATE | IN_MOVED_FROM | IN_DELETE);

to:

#!c++
int wd = inotify_add_watch (mFD, dir.c_str(), IN_CLOSE_WRITE | IN_MOVED_TO | IN_CREATE | IN_MOVED_FROM | IN_DELETE | IN_MODIFY);

For Windows i don't remember quite well, i need to take a look at the documentation, but FILE_NOTIFY_CHANGE_SIZE or FILE_NOTIFY_CHANGE_ATTRIBUTES flags should do the trick ( FileWatcherWin32.cpp line 61 ).

And for OS X i really don't remember how it behaves, but i think that behaves the way you wan't because there's no other way to do it as far i can see from the documentation right now.

Also the Generic File Watcher should work as you want ( generic file watcher is slow but works in the same everywhere, if you are looking less than 50 files shouldn't be a big problem, it can be used initializing with a true as the first parameter of the constructor of FileWatcher efsw::FileWatcher fileWatcher( true );.

The fact that this behaves differently from OS to OS is making me think that i should make this changes in the library, so give me a couples of days to test if i'm right in all that i'm saying, right now i can't test it.

Regards,
Martín

from efsw.

Original comment by Sarvagya Pant (Bitbucket: sarvpant, ).

Hi thanks for the response. The thing is that I don't have control over logger. The logger will be writing into log, which upon modified should trigger efsw. In the FileWatcherWin32.cpp I found you were using FILE_NOTIFY_CHANGE_CREATION, FILE_NOTIFY_CHANGE_LAST_WRITE,FILE_NOTIFY_CHANGE_FILE_NAME,FILE_NOTIFY_CHANGE_DIR_NAME. Since the FILE_NOTIFY_CHANGE_LAST_WRITE doesn't get updated, efsw could not find the change in file. I think I will try adding FILE_NOTIFY_CHANGE_SIZE, FILE_NOTIFY_CHANGE_ATTRIBUTES in the FileWatcherWin32.cpp file. I will ping you in case if that works.

from efsw.

Original comment by Sarvagya Pant (Bitbucket: sarvpant, ).

Hi, I updated efsw with those additional filter conditions. It could detect Modified if attributes of file gets changed but cannot detect any change in the log file size.

FILE_NOTIFY_CHANGE_SIZE

The operating system detects a change in file size only when the file is written to the disk. For operating systems that use extensive caching, detection occurs only when the cache is sufficiently flushed.

Is this problem windows related? Because the doc says that the last_write and size will trigger change only when the file contents are flushed. If this is the case, does this mean for loggers out there, efsw could not trigger the change. The file handle may not be closed by the logger. Is this causing last_write and size not to get updated? But once the file gets opened by external program, they get updated. Any idea on this issue.

from efsw.

Original comment by Martín Lucas Golini (Bitbucket: SpartanJ, GitHub: SpartanJ).

I'll have to take a look.

Usually the flush happens often, it shouldn't be an issue.
I'm not sure if i understood what you said, the FILE_NOTIFY_CHANGE_SIZE should happen every time the cache is flushed, FILE_NOTIFY_CHANGE_ATTRIBUTES should happen also every time that the cache is flushed because the modification time of the file changes.

You said It could detect if attributes of file gets changed but cannot detect any change in the log file..

I don't get it, what's the difference?

The file handle may not be closed by the logger. Is this causing last_write and size not to get updated?

It shouldn't. The size change should be independent of the file descriptor close.

But once the file gets opened by external program, they get updated. Any idea on this issue.

When you open the file the last access attribute will be updated and it will trigger the FILE_NOTIFY_CHANGE_ATTRIBUTES.

from efsw.

Original comment by Sarvagya Pant (Bitbucket: sarvpant, ).

Hi, I tried to create a short version of code that could capture the whole scenario. I have created a video of the process. Please check here. I have created a logger that writes in file. I used Log4Cplus. Check the logger code here .
If you watch during the end of video, when I click or open the file only then efsw triggers Modified. If this is a problem of Windows OS, how often does Windows flush the contents. If its efsw how should I handle that scenario.
Thank you.

from efsw.

Original comment by Martín Lucas Golini (Bitbucket: SpartanJ, GitHub: SpartanJ).

Hi!
I've been testing a little bit and it seems that the Windows file system cache is really weird. The FS cache is big, i took me like 200 MB to start seeing the modified events, i really didn't know about this.
I tried in Linux adding the change that i mentioned before and it works fine ( you receive constant Modified events ).
I'll continue investigating. There must be a way to force the cache write to disk like the file explorer does ( because that's what it's happening, it's manually forcing this ).

Regards

from efsw.

Original comment by Sarvagya Pant (Bitbucket: sarvpant, ).

Hi. Thank you for the response. It seems it is issue of Windows OS on flushing the contents in file rather than efsw. If there is some way to force the cache write, please share here too. Thank you.

from efsw.

Original comment by Martín Lucas Golini (Bitbucket: SpartanJ, GitHub: SpartanJ).

Yes! Sadly is a windows thing, i'm still looking for a real solution.

May be for your use case there's a partial solution to this. The thing is that you can force to flush the cache opening the file with the flag FILE_FLAG_NO_BUFFERING. But in that case efsw shouldn't be necessary at all, you could just compare the size and modification time of the file from time to time, but well... efsw would do that for you too.

I'm impressed by the little documentation that there is about this, from Microsoft i only found this.

Let me know if you find something, i'll close the bug because somehow is invalid, not a problem of the library, but i'll continue looking for solutions.

Regards and thanks for your help.

from efsw.