autozimu / unison-fsmonitor Goto Github PK

View Code? Open in Web Editor NEW

73.0 4.0 9.0 67 KB

unison-fsmonitor implementation

License: MIT License

Rust 96.20% Nix 3.80%

unison-fsmonitor macos unison

unison-fsmonitor's Introduction

unison-fsmonitor

Why

unison doesn't include unison-fsmonitor for macOS, thus -repeat watch option doesn't work out of the box. This utility fills the gap. This implementation was originally made for macOS but shall work on other platforms as well like Linux, Windows.

Install

brew install autozimu/homebrew-formulas/unison-fsmonitor

Alternatively if you have cargo installed,

cargo install unison-fsmonitor

Usage

Simply run unison with -repeat watch as argument or repeat=watch in config file.

File watch limits

You might need to update file watch limits in both hosts if watching limit reached. See https://facebook.github.io/watchman/docs/install#system-specific-preparation for more details.

Debug

RUST_LOG=debug unison

References

unison-fsmonitor's People

Contributors

Stargazers

Watchers

Forkers

seeekr bauerxcel icodein offbyone ejuha dylan-dpc porelli alinstoian thomas-ross-aws

unison-fsmonitor's Issues

Apple Silicone

Hello! We're successfully using unison in fairly high volume with a macOS M1 binary we've compiled. This is a request to see if you can compile your rust project on an M1 and perhaps update the homebrew release.

If you don't want to, it would be nice to get a copy of the entire rust project so we can give it a go.

Big thanks :)

"No space left on device" error coming from remote monitor

I just upgraded to your 0.3.0 release, and I'm no longer able to have unison working. On the local side, I'm getting this error:

2022-03-11T19:06:28Z DEBUG unison_fsmonitor] event: Input("DONE\n")
Error: Io(Os { code: 28, kind: StorageFull, message: "No space left on device" })
Fatal error: Server: File monitoring helper program not running
[2022-03-11T19:07:06Z DEBUG unison_fsmonitor] event: Input("")
[2022-03-11T19:07:06Z DEBUG unison_fsmonitor] >> ERROR Unrecognized%20cmd%3A%20

I believe that this is being emitted by the unison-fsmonitor on my remote host, which is being built from the current tip of your master branch.

I am not, as far as I can tell, out of space, either locally or remotely.

$ df -i
Filesystem       Inodes    IUsed    IFree IUse% Mounted on
devtmpfs       24200302      542 24199760    1% /dev
tmpfs          24202461        2 24202459    1% /dev/shm
tmpfs          24202461      707 24201754    1% /run
tmpfs          24202461       16 24202445    1% /sys/fs/cgroup
/dev/nvme0n1p1 31457280 17799544 13657736   57% /
tmpfs          24202461        5 24202456    1% /run/user/272766
tmpfs          24202461        1 24202460    1% /run/user/0
tmpfs          24202461        1 24202460    1% /run/user/221009

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         93G     0   93G   0% /dev
tmpfs            93G     0   93G   0% /dev/shm
tmpfs            93G  2.1M   93G   1% /run
tmpfs            93G     0   93G   0% /sys/fs/cgroup
/dev/nvme0n1p1  473G  312G  161G  66% /
tmpfs            19G     0   19G   0% /run/user/272766
tmpfs            19G     0   19G   0% /run/user/0
tmpfs            19G     0   19G   0% /run/user/221009

(all of this is in a directory on the / mount)

I've "worked around" this by switching to a different fsmonitor, which is ... okay, I guess, but yours has been the most consistently stable of the ones I've used so far, and I would prefer to keep using it if I could. I'm not sure how to debug this on the server side, though.

Can't sync to my server due to Filesystem watcher error

Hi,

I'm seeing following error

Fatal error: Server: Filesystem watcher error: cannot add a watcher: system limit reached
The watcher can be disabled by setting preference 'watch' to false

Any suggestion please? Thank you.

Support for Homebrew on Linux (amd64 and aarch64)

Similar to #12

When adding the formula and installing the package, the system receives the MacOS binary that of course cannot be executed.

$ file /home/linuxbrew/.linuxbrew/Cellar/unison-fsmonitor/0.3.0/bin/unison-fsmonitor
/home/linuxbrew/.linuxbrew/Cellar/unison-fsmonitor/0.3.0/bin/unison-fsmonitor: Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE|HAS_TLV_DESCRIPTORS>

Do you have any plan to support different platforms/architecture?

Thanks!

invalid attribute for formula 'autozimu/formulas/unison-fsmonitor': version (nil)

Running:
brew update
brew upgrade
brew doctor <- all good
brew install autozimu/homebrew-formulas/unison-fsmonitor

I got:

Updating Homebrew...
==> Tapping autozimu/formulas
Cloning into '/usr/local/Homebrew/Library/Taps/autozimu/homebrew-formulas'...
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 16 (delta 1), reused 8 (delta 1), pack-reused 8
Unpacking objects: 100% (16/16), done.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/autozimu/homebrew-formulas/Formula/unison-fsmonitor.rb
invalid attribute for formula 'autozimu/formulas/unison-fsmonitor': version (nil)
Error: Cannot tap autozimu/formulas: invalid syntax in tap!

Stale cargo / brew packages

There have been two pretty important bug fixes in the past two weeks[0][1]. This package wasn't usable for me before these fixes and I presume for lots of other people. I was able to build locally and it worked great. Any chance of having new packages published as a new patch version?

Thank you for the great contribution!

0: f37c270
1: 485b6e6

"Invalid argument: unquote" from lwt.ml, line 126, characters 16-23

My unison-script is crashing with the following output:

Looking for changes
Invalid argument: unquote
Raised at Lwt.poll in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/lwt/lwt.ml", line 126, characters 16-23
Called from Lwt_unix_impl.run in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/lwt/generic/lwt_unix_impl.ml", line 102, characters 8-23
Called from Uitext.Body.synchronizeOnce in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/uitext.ml", line 1124, characters 16-56
Called from Uitext.Body.synchronizePathsFromFilesystemWatcher.loop in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/uitext.ml", line 1183, characters 6-125
Called from Uitext.Body.synchronizeUntilDone in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/uitext.ml", line 1231, characters 8-47
Called from Uitext.Body.start in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/uitext.ml", line 1417, characters 21-43

More info and a case of how to reproduce can be found at the unison repo: bcpierce00/unison#338 (comment)

According to the maintainers there:

The '%' character is not percent-encoded and this is what's causing the error.

I misread the source. Unison does encode the percent character.
This is a bug in the external fsmonitor.

I'm not that familiar with Rust ... if you would have time to confirm the issue and fix it, I would really appreciate it!

Please add a license

This looks like a great tool, and fills an important gap in Unison, but it makes it difficult for people to use it if there is no license.

Check out https://choosealicense.com/ for help picking one. Thanks!

Permission denied error on unison fsmonitor

Hi,

I did sudo chown -R $whoami . in both my local machine and remote machine. But still didn't work.

I also did sudo sysctl kern.maxfiles=524288 kern.maxfilesperproc=262144 to increase the limit.

Could you take a look and help me find out what's going on?

Thanks

Race condition can leave files out of sync while syncing between local dirs

To reproduce (I'm on macOS 12.3.1):

mkdir /tmp/a /tmp/b
unison /tmp/a /tmp/b -repeat watch -ui text -debug fswatch+

While this is running, try

echo $RANDOM > /tmp/a/x

I see a bunch of output, quoted below. Importantly, there is a delay between "Synchronization complete" being printed and new WAIT commands being sent. If you modify /tmp/a/x again during this interval, a second synchronization does not occur, and so /tmp/a/x and /tmp/b/x stay out of sync (until one of them is changed yet again or Unison is restarted).

I can reproduce this on Unison 2.51.5 as well as the latest master at time of writing. It doesn't occur when using Unison's built-in fsmonitor implementation on Linux.

The cause appears to be the timing of CHANGES responses. This fsmonitor implementation sends CHANGES immediately as soon as changes occur. But the built-in fsmonitor has more of a polling model, where CHANGES responses are sent only in response to WAIT commands; a CHANGE response terminates a wait, and changes that occur while not waiting should be reported at the next wait. In fact, Unison itself seems to ignore the arguments to CHANGES responses; it simply uses CHANGES as a notification that it should terminate the wait and send its own CHANGES command to request filenames of the files that changed. (Nevertheless, the built-in fsmonitor does include the hashes of changed files in its CHANGES response; there can be multiple hashes in a single response.)

Edit: To further clarify: WAIT is like select(), terminated by a CHANGES response, and CHANGES commands are like read(). They affect the same state (list of pending changes). I think.

[fswatch+] >> CHANGES d03c239c5bae8b7a9a35a6284d8a9512
Looking for changes
[fswatch+] << CHANGES d03c239c5bae8b7a9a35a6284d8a9512
[fswatch+] >> RECURSIVE x
[fswatch+] >> DONE
[fswatch+] << START d03c239c5bae8b7a9a35a6284d8a9512 /private/tmp/a x
[fswatch+] >> OK
[fswatch+] << DONE
[fswatch+] << CHANGES ee6f81089ac5b4e616740c34d688212d
[fswatch+] >> RECURSIVE x
[fswatch+] >> RECURSIVE %2Eunison%2Ex%2Eee6f81089ac5b4e616740c34d688212d%2Eunison%2Etmp
[fswatch+] >> DONE
[fswatch+] << START ee6f81089ac5b4e616740c34d688212d /private/tmp/b x
[fswatch+] >> OK
[fswatch+] << DONE
Reconciling changes
changed  ---->            x  
a            : changed file       modified on 2022-05-09 at 19:05:43  size 6         rw-r--r--
b            : unchanged file     modified on 2022-05-09 at 19:05:39  size 6         rw-r--r--
Propagating updates
Unison 2.51.5 (ocaml 4.12.1) started propagating changes at 19:05:44.67 on 09 May 2022
[BGN] Updating file x from /private/tmp/a to /private/tmp/b
[END] Updating file x
Unison 2.51.5 (ocaml 4.12.1) finished propagating changes at 19:05:44.67 on 09 May 2022, 0.001 s
Saving synchronizer state
Synchronization complete at 19:05:44  (1 item transferred, 0 skipped, 0 failed)
[fswatch+] >> CHANGES ee6f81089ac5b4e616740c34d688212d
[fswatch+] >> CHANGES ee6f81089ac5b4e616740c34d688212d
[fswatch+] >> CHANGES ee6f81089ac5b4e616740c34d688212d
[fswatch+] << WAIT d03c239c5bae8b7a9a35a6284d8a9512
[fswatch+] << WAIT ee6f81089ac5b4e616740c34d688212d

unison crash after first sync with No file monitoring helper program found

unison -version                                   
unison version 2.51.2 (ocaml 4.08.1)

unison config

root = /Volumes/DevDesktop
root = ssh://[email protected]//workplace/mbazzan
ignore = Name .FBCIndex
ignore = Name .FBCLockFolder
ignore = Name .fseventsd
ignore = Name {.Trash*,.VolumeIcon.icns,.HSicon,TemporaryItems,.Temporary*,TheFindByContentFolder}
ignore = Name .DocumentRevisions-*
ignore = Name {TheVolumeSettingsFolder,.Metadata,.filler,.idsff,.Spotlight,.DS_Store,.CFUserTextEncoding}
ignore = Name */src/*/eclipse-bin
ignore = Name */AAA/*
ignore = Name */.project
ignore = Name */.settings
ignore = Name */.classpath
ignore = Name *.swp
ignore = Name */env
ignore = Name */RemoteSystemsTempFiles
ignore = Name */.idea
ignore = Name */src/*/*.iml
ignore = Name export
ignore = Path moonfish
ignore    = Path */build/*/*/*/*/build/{,.}?*
ignorenot = Path */build/*/*/*/*/build/unit-tests
ignore = Regex .*/runtime/perl/lib/perl...-dist/[pP]od
repeat = 5
maxbackups = 0
retry = 1
auto = true
batch = true
confirmbigdeletes = true
times = true
prefer = newer
terse = true
contactquietly = true
log = true
logfile = /Users/mbazzan/.unison/unison.log
ignorecase = false
fastcheck = true
sshargs = -C

command:
RUST_LOG=debug unison -repeat watch

output

DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: event: Input("VERSION 1\n")
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: >> VERSION 1
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: event: Input("START 2fa3d0d616137cebf6df2910a9a50132 /Volumes/DevDesktop \n")
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: replicas: {"2fa3d0d616137cebf6df2910a9a50132": Replica { root: "/Volumes/DevDesktop", paths: {"/Volumes/DevDesktop"}, pending_changes: {} }}
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: >> OK
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: event: Input("DIR \n")
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: >> OK
...
...
...
...
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
[END] Copying BluTraceBuilderTools/build
[END] Copying BluTraceBuilderTools/logs
[END] Updating file BluTraceBuilderTools/packageInfo
[END] Copying BluTraceBuilderTools/src/BluTraceBuilderTools/.coverage
[END] Copying BluTraceBuilderTools/src/BluTraceBuilderTools/.pytest_cache
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/doc/.unison._apidoc.2fa3d0d616137cebf6df2910a9a50132.unison.tmp"), op: Ok(CHMOD | CREATE | RENAME), cookie: Some(89893770) })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/doc/_apidoc"), op: Ok(RENAME), cookie: Some(89893770) })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/src/blu_trace_builder_tools/.unison.extracommands.py.2fa3d0d616137cebf6df2910a9a50132.unison.tmp"), op: Ok(CHMOD | CREATE | RENAME | WRITE), cookie: None })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/src/blu_trace_builder_tools/extracommands.py"), op: Ok(RENAME), cookie: None })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/test_profile/fsts/development/fst_data/Video_BrowseVideoIntent/.unison..build.manifest.2fa3d0d616137cebf6df2910a9a50132.unison.tmp"), op: Ok(CHMOD | CREATE | RENAME | WRITE), cookie: Some(89893779) })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/test_profile/fsts/development/fst_data/Video_BrowseVideoIntent/.build.manifest"), op: Ok(RENAME), cookie: Some(89893779) })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
[END] Copying BluTraceBuilderTools/src/BluTraceBuilderTools/doc/_apidoc
[END] Updating file BluTraceBuilderTools/src/BluTraceBuilderTools/src/blu_trace_builder_tools/extracommands.py
[END] Copying BluTraceBuilderTools/src/BluTraceBuilderTools/test_profile/fsts/development/fst_data/Video_BrowseVideoIntent/.build.manifest
Synchronization complete at 11:13:26  (12 items transferred, 0 skipped, 0 failed)
DEBUG 2020-06-24T09:13:27Z: unison_fsmonitor: event: Input("WAIT 2fa3d0d616137cebf6df2910a9a50132\n")
Fatal error: Server: No file monitoring helper program found
DEBUG 2020-06-24T09:13:27Z: unison_fsmonitor: event: Input("")
DEBUG 2020-06-24T09:13:27Z: unison_fsmonitor: >> ERROR Unrecognized cmd:

and it exits

Add documentation: Unix.EBADF with unison / unison-fsmonitor - change /proc/sys/fs/inotify/max_user_instances

especially when running Unison, I quite often experienced the following error:

Fatal error: exception Unix.Unix_error(Unix.EBADF, "set_nonblock", "")

Running "strace unison-fsmonitor" yielded:

fcntl(0, F_GETFL)                       = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(0, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
fcntl(1, F_GETFL)                       = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(1, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
inotify_init()                          = -1 EMFILE (Too many open files)
fcntl(-1, F_GETFL)                      = -1 EBADF (Bad file descriptor)
write(2, "Fatal error: exception Unix.Unix"..., 71Fatal error: exception Unix.Unix_error(Unix.EBADF, "set_nonblock", "")) = 71
exit_group(2)                           = ?
+++ exited with 2 +++

I have fiddled around with 'ulimit -n', without lasting success.

The real solution was to change the following:

sysctl -w fs.inotify.max_user_instances=256 >> /etc/sysctl.d/unison.conf

The original OS vendor setting (OpenSuSE) was at 128. And with Nextcloud and another sync tool running besides Unison my system seemed to have to many inotify resources used.

I would appreciate if you could add a hint to your documentation somewhere, that inotify limits may have to be increased if the before mentioned error is encountered. The issue may be related to #8 .

Broken pipe [write()]

I'm trying to setup unison to sync a folder between local linux and remote macOS. I installed your utility on macOS, and get this error when running unison from linux with repeat = watch:

Fatal error: Server: Error in sending command to filesystem watcher:
Broken pipe [write()]

Do you know what is causing this?

unison: 2.51.2
macOS: 10.12.6