Inotify support about syncthing HOT 116 CLOSED

The other advantage of this would probably be reduced power usage.

from syncthing.

Yep.

from syncthing.

Btw, this is on top of the todo list now.

from syncthing.

Great!

from syncthing.

This turns out to be not doable using the currently (or soon) available APIs in Go. See http://goo.gl/MrYxyA for some discussion of the problems; in short it doesn't scale on any of the BSDs, including Mac. Possibly something non portable could be implemented for Mac (fsevents), Linux and Windows separately but that's a significant amount of work. Hopefully someone else will do that and package it up. :) Until then, this will have to be put on the back burner.

from syncthing.

Okay. Back burner for Linux too?

from syncthing.

I wonder if it's worth using https://github.com/howeyc/fsnotify in the meanwhile?

from syncthing.

It suffers from the same problem - not using the recursiveness support that the OS might have. This means that for a directory structure like

dir/
dir/file
dir/dir_b/
dir/dir_b/file
dir/dir_c/
dir/dir_c/file

we need to put watches on (and consume one file descriptor per) dir, dir/dir_b and dir/dir_c to catch changes to the files. That doesn't scale.

from syncthing.

For example, on my Mac the default fd limit is 256, and the number of directories in my directory to sync is significantly higher;

jb@jborg-mbp:~ $ ulimit -n
256
jb@jborg-mbp:~ $ find -L Sync -type d | wc -l
    2424
jb@jborg-mbp:~ $ 

The limit can of course be raised, but there's a hard max somewhere and the exact limits are system specific. It sometimes working, sometimes not working and sometimes working to start with but then not working later would not be awesome behavior.

from syncthing.

How about syncthing let something else do the watching, then accept events from that? inotifywatch is a binary that will monitor a directory recursively. Perhaps the output could be parsed or it could be setup to pass the information to syncthing so that the inotify stuff stays separate?

from syncthing.

Inotifywatch is a Linux thing.

from syncthing.

Yes, it's just an example of an external program that feeds data to syncthing.

from syncthing.

Actually, no, that doesn't work either. inotify isn't recursive either, so has the same scalability problems.

jb@syncer:/data/syncer$ inotifywatch -r sync/
Establishing watches...
Failed to watch sync/; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.

from syncthing.

What we could do is use inotify/kqueue/etc for the case where there is only a handful of directories to sync, then fall back to scanning when it doesn't work any more. Personally I think that's pretty crappy and the only gain is to discover changes faster than the scanning interval -- the CPU load of doing the scanning is anyway pretty negligible since we've established there are only a handful of directories to start with...

from syncthing.

I'd say that's a reasonable compromise, and one could expose the directory limit for users who have raised their system's limits and want to push it further (and/or know what they're doing).

from syncthing.

Yeah, I think switching behaviour is fine as long as you tell the user when you do this, otherwise they will see a sudden degradance in the functionality of syncthing and not know what has happened.

from syncthing.

inotify support could be useful for rapidly changing files, it would avoid the delay.

from syncthing.

Hi I think that would be easier on linux systems stay with fanotify ( http://lwn.net/Articles/339253/ ), usually Anti-Virus implementations uses fanotify because you won't need a fd per directory or even create new inotify handlers for new folders on the fly. There's a good implementation of fanotify for GO ( https://godoc.org/bitbucket.org/madmo/fanotify ) I am using this one to scan uploaded files using clamav.

from syncthing.

Another tool that has already solved this problem & has implemented a protocol is Facebook's watchman so perhaps including support for that would be beneficial?

from syncthing.

Indeed, that could be a solution.

from syncthing.

Until there is a better solution inotify or something else: What's about to add a button "scan" in the WebGUI on every repository? Maybe beside "edit" ?

from syncthing.

See http://discourse.syncthing.net/t/manual-rescan-button/34

from syncthing.

I may be confused, but my reading of the inotify API suggests that it does not need a separate file descriptor for each directory. As far as I can tell, you can associate a whole list of watches against a single inotify instance and therefore a single descriptor. It is true that you need to add each subdirectory explicitly, however.

IMHO, syncthing should absolutely be using this feature, where available, especially where there are large numbers of directories to scan. Anything else is just going to annoy the hell out of me.

from syncthing.

It varies per platform. On Linux, as far as I can see, you're correct and this is governed by a separate /proc/sys/fs/inotify/max_user_watches limit. On BSD:s, it's governed by the file descriptor limit (which is usually a lot harsher than what you can max out the linux watches to). On Windows, I have no clue.

from syncthing.

For Python there exist watchdog, a platform independent solution, see: https://pypi.python.org/pypi/watchdog

quote: Supported Platforms:

Linux 2.6 (inotify)
Mac OS X (FSEvents, kqueue)
FreeBSD/BSD (kqueue)
Windows (ReadDirectoryChangesW with I/O completion ports; ReadDirectoryChangesW worker threads)
OS-independent (polling the disk for directory snapshots and comparing them periodically; slow and not recommended)

There is also a note of the FreeBSD/BSD kqueue limits, see: https://github.com/gorakhargosh/watchdog#supported-platforms

EDIT: Found this Go project: https://github.com/howeyc/fsnotify

from syncthing.

May I suggest a solution: add inotify/kqueue/etc. watchers to the most-recently-changed directories until the limit is reached. Then scan the remaining directories the hard way. When a non-watched directory is changed, replace the least-recently-used existing watch with the new one. Naturally, the user should be notified, in some suitably non-obnoxious way, that the limit has been reached (and how to increase the limit, if appropriate).

from syncthing.

Guys... You're not adding new information. Look back at the posts from january and march and the links. The point is not that it's impossible, or that there is nothing cross platform. It is that the cross platform stuff doesn't scale, and the stuff that does scale is not cross platform. As far as I can tell there exists solutions that will work for Linux (requiring the user to tweak their stuff in /proc), Mac (entirely different API) and Windows (again, different API). For other BSD:s, there seems to exist nothing that scales.

Implementing it is certainly possible, but it's a significant amount of work, will require falling back to the current algorithm in a bunch of cases, and adds no significant value compared to a lot of other stuff that needs to be done.

from syncthing.

Significant for whom? This is a deal-breaker for me. It might be that syncthing is not the right solution to my problem, but it looks like it could be perfect, with this feature. I've not found anything else. I could maybe glue inotifywatch to rsync with an expect script, or something, but I was hoping for something less hacky. Anyway, sorry to have bothered you.

from syncthing.

@a-m-s Now I'm curious. How is this a deal breaker? What significant visible change in syncthings behavior do you expect to see from this?

from syncthing.

Sync will be faster, because filesystem changes must not wait for "Rescan Interval"... And rescaning the filesystem is slower.

But it seems that there is no usable cross-platform solution for recursive watcher out there. see:

• howeyc/fsnotify#56
• howeyc/fsnotify#89

from syncthing.

Yes, it will be slightly faster in some circumstances, although in most cases I think the time is dominated by the actual time for other nodes to sync and not necessarily detecting the change. And we'll anyway have to delay the notifications a while to "debounce" and increase the chance that the file has stopped changing for he moment. But still. Is this the deal breaker?

from syncthing.

OK, here's my use case:

I have a large source base that must be built on a remote server. There are approximately 80000 files spread across 5000 directories, and I typically have 4 or five of these on the go at any one time. The remote server's ping time is just a little too high to edit files comfortably, and anyway I'd rather use a nice editor with indexing etc., such as sublime, or eclipse, or whatever. The ping is also too high to use SSHFS with these editors (they freeze for a few seconds at regular intervals), and anyway doing file searches is not practical this way.

So, I want a file-sync that will allow me to mirror my sources locally, edit & search them quickly, and be able to build remotely as soon as I save an edit locally (e.g. hit build all in eclipse, with that set to run ssh svr make). My working set is typically spread over only a few directories, on any given day, and the file delta will typically be small, so as long as the synchronization happens promptly then network bandwidth ought to be no problem. "Bouncing" is not a bit deal either.

Anyway, since writing my previous comments I kept searching, and I've now found lsyncd, and this seems to do the job, with -delay 0. It also has the advantage that it requires nothing more than ssh and rsync on the remote server.

from syncthing.

@a-m-s Ah, I see. Yes, that makes sense. It's not a use case enormously well suited to syncthing right now. Hopefully we can get there at some point.

from syncthing.

@a-m-s I think git would be more suited to that use case ;)

from syncthing.

@rom1504 Do you have it automated somehow? It's not a win if I have to run git commands every time I save a file. Nor would it win if it has to scan the entire tree every time anything changes. Having my edit history saved would be an interesting feature, as would edit conflict detection. Sure I can imagine how I might go and script up something that works, but then I wouldn't be doing real work.

from syncthing.

@a-m-s you can have a git hook that trigger the build each time you push

from syncthing.

@rom1504 that "fixes" the wrong end of the problem.

from syncthing.

Okay here's my idea for this:

Add a function to the Rest API like POST /rest/file_update with params type (create/edit/delete/move) and path (or path_from and path_to for move).

This would go with a config.xml per-repo setting scanner_enabled (true by default).

Then a seperate, host-specific program (Android client, Desktop program thumbnail etc) could receive inotify events, and send them to the syncthing binary.

@calmh What do you think about this, and if you like it, could you leave it for me to implement? :)

from syncthing.

Awesome idea. I hadn't noticed this when I first saw #383. This seems to allow for the most flexibility given the constraints mentioned above.

In practice, would the external scanner, if enabled, completely replace the internal scans as they are done now? The reason I ask is that if the external scanner fails, syncthing has no way of detecting this without monitoring the external process itself.

from syncthing.

Yes it would be replaced (as it's also supposed to save energy on Android).The external scanner failing should never happen.

from syncthing.

I think this is a nice way to hook in. It should be implemented.

from syncthing.

will do ;)

from syncthing.

@bsidhom If the scanner process is launched by the main process then a failure can be easily detected via a SIGCHILD handler (on *nix). If that's not available, then checking the PID once a minute is significantly quicker,and less battery hungry than scanning a large directory structure. If the scanner is considered stable, then the PID check could be reduced to once per hour, or per day, or whatever.

from syncthing.

Yeah, even scanning for the notifier every so often should consume far less power than a full directory scan every minute. I'm just not sure if it's advisable to have the syncthing daemon actually launch the listener. Requiring as much would prevent the user from attaching a file system notifier that was already in use to syncthing. It's conceivable that a repository might have its own file change hooks.

from syncthing.

It's conceivable that a repository might have its own file change hooks.

Conceivable, but probably not the common case. Wouldn't that be an optimisation though? The kernel should handle that.

from syncthing.

I would argue that external users of the API are not syncthing's problem. The 'built-in' scanner, whatever its implementation, is syncthing's problem.

from syncthing.

Yeah, I think it should work so that an external program launches syncthing, feeds in any inotify events via API and acts as a system service.

Syncthing basically shouldn't even know it is there. And any bugs in the external program really aren't a concern for syncthing.

from syncthing.

@Nutomic , what is the status of the REST API? It's very critical to my use-case where I am accepting files via SMTP attachments over TLSA DANE encryption, and depositing them to syncthing repo, based on the recipient email address. It makes for a very good solution to those "I just received an email on mobile and wanna forward this to my repo" moments.

from syncthing.

Maybe most people didn't get what I said earlier. In short, it just means that you no longer have to rely on "Save to My Google Drive" on Gmail. Just forward the email securely to the syncthing server, and work on your documents and creative assets when you get back.

from syncthing.

@heri16 I'm not sure you got the purpose of the proposed API right. What I want to implement is only for inotify support. So you have an external program that monitors the syncthing directory for changes, and when it detects a change, sends info about it to syncthing via the API.

What you want sounds more like a web interface (accessing syncthing files from another device where syncthing isn't installed).

That said, I'm rather busy at the moment so I haven't started yet, hope I'll get around to it soon.

from syncthing.

@Nutomic I'm not sure that you got me right. An API is an API. Once integrated with additional tools, it can be used for many different things. That's why it's an API. There is no mention of web interface, at all. I'm not sure where you got that from. There is only an API to make Syncthing realize immediately that a new file has just been written to. This API is called by the external Email MIME parser. http://mailin.io

from syncthing.

@heri16 The API you describe may or may not be useful, but is certainly off-topic for this thread. inotify support requires only that syncthing is informed that a particular existing file has had activity. It does not pass any user data. It does not create any files (although it might occur in response to a file being created).

from syncthing.

@ams-cs I am just explaining one of many use cases of the inotify API. How would this be off-topic? I had to elaborate on this use-case, despite not wanting to initially, to help clarify that syncthing is not passing any user data precisely because that role is left to external tools. Hoping to help other developers here expand their perspective on the value of this API enhancement. Just like how others are using BTSYNC as a low-level layer to host websites and social network apps.

from syncthing.

@heri16 I think I may have misunderstood what you were talking about (and used the wrong account to reply). In any case, there is no need for 3rd Party apps to hit the proposed API. With inotify support your file write will be detected right away, without any action from you, and even without inotify it'll notice it in the next 30 seconds (on average), and that's good enough for most applications.

from syncthing.

@a-m-s The use case for the proposed API is because I don't see a clean and useful way to integrate inotify support in syncthing, as detailed in lots and lots of messages above. The API in question would simply allow a user to rig something that does listen for inotify and tell synchting about changes.

from syncthing.

@calmh That's what I'm trying to explain to @heri16, who seems to think that he needs to poke the API himself.

from syncthing.

Well, he could, to let syncthing know that he created a new file. It doesn't have to be used for inotify notifications if it's just a general "Hey, syncthing! Look at this file/directory right now!".

from syncthing.

Thanks @calmh for coming in. I think @a-m-s still hasn't realized that it has already been decided that Syncthing won't be subscribing to inotify events directly. The "look at this file right now" API call is precisely the cleanest way to implement this, whatever the platform may be.

from syncthing.

This (the external scan API, not inotify support as such) is now implemented. See https://discourse.syncthing.net/t/the-rest-interface/85 for "/rest/scan".

from syncthing.

Shouldn't we keep this issue open? Inotify support is separate from the external scan API, IMO.

from syncthing.

Inotify support is wontfix at the moment. Pull requests or (well thought trough) implementation suggestions are welcome though.

from syncthing.

Somebody could write an inotify stub, and zerotier would detect the platform and auto-download this stub, if the user chooses to do so through the web UI. Syncthing should generate and pass the API key automatically, by launching the stub as an external process. This keeps everything simple for the user, just like one-click install for Firefox or Chrome App Store browser extensions.

from syncthing.

I'd happily write a standalone program that does inotify, but I think the language of choice would be 'go', and I'm not familiar with that. I presume the project already contains/references libraries that can do HTTP client requests?

@heri16 Is your suggestion cross-platform, or would it only work on x86?

from syncthing.

@heri16 sorry, but what is zerotier? Google seems useless, as all I get is VPN providers.

@a-m-s one of the reasons I've started contributing was to learn go a bit more ;)

So it seems this will be part of Go 1.4 or 1.5, meaning that you can fairly easily make it cross platform up to some extent.

It should be fairly easy to do, ideally I'd do a completely separate daemon which given an API key, would notify syncthing of the events.

Since repos can come and go, it should probably poll syncthing via the API for repo changes.
A good start to using syncthing HTTP API with go might be this

from syncthing.

I actually started implementing a Linux-specific one in Rust using these inotify bindings. However, I haven't worked on it for a while because it was difficult to get the recursive watch correct.

Specifically, because Linux doesn't offer recursive inotify at the kernel level, this must be done manually. It's easy to start a watch on the main directory and then recursively add directories below it, but it's difficult to detect when subdirectories have been moved outside of the main directory. This is because the events fed from the kernel itself only contain the last path element of the file for which the event has occurred. The way around this is to create a mapping from watch descriptors to filesystem paths in user-space and then to use this mapping on the watch descriptor received with each notification event. The problem with the linked fsnotify package is that it doesn't return watch descriptors with events, so there's no way to find a fully-resolved file path without creating a new Watcher for each subdirectory and then multiplexing their event streams.

from syncthing.

After skimming through the source code for fsnotify, I've realized that the mapping is maintained by the library itself. Some quick tests show that it doesn't solve the case of a directory being moved out of its parent though.

from syncthing.

Interesting conundrum. A new Watcher for each directory is not an option though: it will soon run out of file descriptors (at least on Linux).

I guess the "safe" thing to do for moved directories is to just tell syncthing to rescan everything. Is it possible to tell where a file moved from?

from syncthing.

‎It doesn't have to be perfect. The Minimum Viable Product is fine.

from syncthing.

@a-m-s: The API doesn't seem to expose anything that would allow this. Unfortunately, I don't see a way to maintain parent information externally without at least some canonical representation of each file object being moved (e.g., a watch descriptor or a (fully-resolved) canonical path name).

from syncthing.

@bsidhom Yes, I think it's just going to have to have a scan-everything fallback for every case where it can't determine exactly what happened.

It ought to be possible to have one Watcher for each repo though, so you'd at least know which one to rescan.

from syncthing.

That's a good idea actually, as there are certain cases where we can definitely determine what happened. My only concern is continuing to trigger full repository scans due to a rogue listener.

For example, suppose that A is our main repository directory and that directory B is a subdirectory of A. If B is moved outside of A, we should stop listening to events from B and all of its subdirectories. However, the watch is not detached automatically, so we will continue getting events. We could try to remove the watch, but doing so requires a path to the directory to stop listening on. Because the event generated when B was moved does not contain a full path, we do not actually know which path was removed.

Note that the Linux API actually distinguishes between move-in and move-out events and between directories and regular files, so we could potentially just remove the listener on any directory that is moved-out and add a listener on any directory that is moved-in. We would seemingly only re-add listeners to the directories that actually fall under the repository in this case, since we're only receiving messages from directories that are already being watched. The problem, though, is nested directories. Suppose that C is a subdirectory of B in the example above. This means that once watches have been established, we have a watch on C itself. When we receive a move-out event for B, we remove the listener, but still maintain listeners on C and all of its descendant directories. How can we mitigate this? We could recursively remove the listener on C and all of its descendants. But then in the case where B is moved back into A, we need not only to add a listener to B, but recursively add all of its descendants as well; we can't depend on a move-in event for C because it was already contained by B. This solves the issues above, though in a fashion that is less efficient than desired. However, we can only do this when we can distinguish between move-in and move-out events and between regular file events and directory events. Check out this man page for a list of event types.

from syncthing.

@bsidhom Have you checked if there's any library for inotify? There must be other projects that need this.

from syncthing.

Has watchman solved this? If they haven't, in 14k lines of C, then the hopes of us doing it in an afternoon are probably slim.

from syncthing.

Sorry, I don't know much about this so:

Why can't we build the tree as we add the listeners?
As we remove a node, we remove all the child nodes.

Once we receive an event from a directory (such as it being moved), why can't we do a stat on the directory on our expected path to see if its still there or not? If not, stop listening.

from syncthing.

OK, I've been learning how to read Go (and maybe even write a little -- http://tour.golang.org/ is cool), and I've looked at the fsnotify code.

It looks like the inotify "cookie" is completely ignored (it reads it, but does not use it for anything). That means the rename (move) events will always show up in pair, if they occur within the tree, but a move in or a move out will show as one event only.

Therefore, when:

• a directory we recognise is moved, then we should automatically unwatch it, and any subdirectories.
  • if the directory was moved away then we don't care about it any more
  • if the directory was moved internally then the other event of the pair will arrive shortly .. see below
  • the fsnotify code is smart enough to associate the old path with the correct descriptor
• a directory we don't recognise appears, then we should add the watches recursively.
  • if the directory was moved in then it's new, and we should watch it
  • if the directory was moved internally then we need to replace the watches we just removed.

In both cases, we should trigger a syncthing rescan of the entire tree below that point to catch the update and any updates that happened while we were futzing with the watches.

from syncthing.

The part I've not checked is whether the events will arrive in pairs on the other OS.

from syncthing.

The cookie trick sounds like it could work really well, though I don't know if the pair of events would be sent consecutively or separated by a nondeterministic string of events. Additionally, from what I recall, all move events contain just the final path element, so it would still be difficult to track down the exact directory. I'll need to re-check this though, because this might just be for move-out events. In any case, it looks like the approach to use will need to be platform-specific.

from syncthing.

Is there currently a way to disable the internal scanner when inotify is used, other than setting the scan interval very high? (does setting it to 0 disable it?)

from syncthing.

@calmh ^

from syncthing.

Setting it to a high value sounds fine to me.

from syncthing.

Alright, although imo there should be a way to disable it completely (that's way cleaner).

from syncthing.

Possibly, now that there is a manual rescan button. Prior to that, it would just have been weird and confusing.

from syncthing.

For the record, fsnotify might come with 1.5, but it's not going to support recursive watches according to the design doc.

I don't understand the scaling argument here. What is the expectancy here? O(log n) in the number of subdirectories + files in them? Memory-wise? CPU time wise?

No, watchman (which uses inotify) does not scale in a magical way (see here). No, os/fsnotify won't do that for you either. There is no magic-wand for inotify, and there never can be (unless inotify API changes someday). Nothing "scales" with inotify, period. You have to face the truth: today, you have to use inotify on Linux, and you have to create a watch for each subdirectory. (Yes, inotify is terrible. Further reading 1 2. Unfortunately, it's the current state-of-the-art watcher on Linux.).

Maybe it doesn't "scale" in some sense, but it is clear that it scales much better than scanning files one-by-one periodically (until you hit the kernel limits, in which case it is user's responsibility to increase the limits --as it is the case with every single inotify client out there). With inotify, at least the userspace complexity will only depend on the number of actual subdir events, not something in the number of subdirs+files periodically.

Waiting for unicorns is just that.

from syncthing.

@raichu Thing is, there are recursive watches at the kernel level on all supported OS:es other than FreeBSD at least Mac OS and Windows. It would be nice to use those, because they would "just work". Lacking that it's going to have to be a hybrid approach of watching some directories and scanning the rest - just failing and asking the user to tweak some kernel parameter is not acceptable. That hybrid approach has some advantages, but is much much more complex than what we have today. I don't like the tradeoff enough to try writing it. Someone else is welcome to. :)

from syncthing.

I've just created a repository here: https://github.com/a-m-s/syncthing-inotify

This is an initial implementation of a recursive fsnotify wrapper. It doesn't do anything as a standalone program (yet), but you can run "go test".

I need to add the HTTP stuff to interface with syncthing, of course, and a suitable main function to start and stop it.

This is the first time I've written any Go, so apologies if it looks a bit odd.

from syncthing.

@a-m-s If it's in Go, why not merge it directly into the syncthing code?

from syncthing.

@Nutomic That would be the end goal. If you have any suggestions about the proper way to do that then go ahead.

from syncthing.

Well I would probably do a pull request, or make it a library. That way you avoid the hassle of HTTP calls.

from syncthing.

@a-m-s Obviously, if it works well, I'll be more than happy to merge it. That'll probably make it easier to handle some of the nasty conditions as well (such as we getting notifications on all our own changes while syncing).

from syncthing.

It is probably still easier to do all that testing and playing around outside of syncthing "core" though. When that works, merging it into core is probably not too difficult.

from syncthing.

I made it work using the REST interface, see https://github.com/Zillode/syncthing-inotify. Tested on OSX and Windows.
However:

• I get "too many open files" errors when using big repositories. After modifying the ulimits, my OSX system became unstable (cannot open new applications, extremely slow, ...).
• The watcher sometimes triggers multiple times when files are duplicated. When directories are removed, all contained files also cause a trigger.

from syncthing.

@Zillode I can test on Linux. I can't seem to create an issue there, so will report here: I'm not sure the binary is doing anything (all it says is "Press enter key to exit") - how do I check that - can it show me info about which repos it's doing something with, or a verbose mode or something? Also a warning about increasing interval of/disabling manual repo scan would be nice.

from syncthing.

@Zillode nice!

The "too many open files" thing should not happen on Linux, if fsnotify is coded correctly, but maybe this will always be a problem on OSX?

I've always expected that a complete implementation would include a small delay to aggregate multiple events into one, so rm -rf dir will result in only one notification to syncthing. Obviously, this is not necessary in version 0.01. :-)

from syncthing.

I opened the issues tab on my repo and added some logging output. It seems to be an OSX/BSD specific issue indeed (howeyc/fsnotify#54)

from syncthing.

That's funny, I've made a shell script just for this and went here to post it and I'm late by 20 hours. Anyway, maybe it will be useful. Parameters are set in stwatch.conf which should be placed to the same directory where the script is located. It may look like this:

APIKEY=SDFJHLEURH121412W2488HRHFF984FD
ADDRESS=192.168.1.2:8080
WATCHDIRS=(~/repo1 ~/another_repo)
WATCHREPOS=(repoid1 repoid2)

The only requirements are bash and inotify-tools. You can also override the timeout, TIMEOUTCHANGE defines the "cooldown" after change is detected. Only if nothing else changes in that number of seconds, the repo is getting updated or else it waits another 5 seconds (by default). This is to prevent many updates on file copying or another bulk operation.

from syncthing.

@rkfg I tweaked your script to work on cygwin with https://github.com/thekid/inotify-win. That version of inotifywait has no timeout setting so I had implement a workaround.

https://gist.github.com/parity3/6e0bd4979d9e808ecb86

I'm in agreement with above comments that path recursive change monitoring is definitely in this project's core scope. No matter how non-cross-platform, multi-faceted, difficult, or performance hungry, and no matter how much build complexity it adds, this feature needs to implemented in some fashion and bundled not as a plugin but built into the program. The good news is that this is largely independent of the rest of the program's inner workings.

'All we need' is someone proficient in integrating all existing OS file notification mechanisms so they are accessible in GO. If that's not possible, then start with these scripts and just manage the off-shoot process as a quick-and-dirty solution (that can be disabled via check box in folder config, right next to the scheduled sync time setting).

For me, I it meant a longer install time. For someone else, they've already moved on.

from syncthing.

@parity3 I agree that this is a must have feature. AFAIK it's the only way to let hard drives go to sleep mode while syncthing is running.
That's an important feature for users that have a NAS running 24/7 and still want to save some energy.

from syncthing.

I also wanted to chime in that this is a deal breaker for me and I consider myself a typical use-case. Often you're working on a document on your laptop or PC, save and then shutdown to run to a meeting/event/etc - presently you have to wait a few minutes before shutting down. A major inconvenience, more so on laptops.

Also, whilst I don't have the data, I can't imagine scanning every 60 seconds is particularly battery friendly.

from syncthing.

FYI, no longer using the bash script I posted earlier. syncthing-inotify has evolved enough to be usable on windows. I don't see heavy resource usage so it seems to be using native fs monitoring hooks. It also handles telling syncthing specific files to rescan, but currently the documentation suggests to not fully unset rescan, but instead set it to 86400 (daily). Good enough for me.

from syncthing.

Scanning every 60 seconds is not the most efficient thing it could do, but since that data is almost certainly memory cached, it's not quite as painful as it could be.

inotify is certainly an improvement, but it is possible to confuse it slightly (by moving files over the top of other files, etc), so yes, a daily watch-reset and rescan wouldn't do any harm.

from syncthing.