autozimu / unison-fsmonitor Goto Github PK
View Code? Open in Web Editor NEWunison-fsmonitor implementation
License: MIT License
unison-fsmonitor implementation
License: MIT License
I just upgraded to your 0.3.0 release, and I'm no longer able to have unison working. On the local side, I'm getting this error:
2022-03-11T19:06:28Z DEBUG unison_fsmonitor] event: Input("DONE\n")
Error: Io(Os { code: 28, kind: StorageFull, message: "No space left on device" })
Fatal error: Server: File monitoring helper program not running
[2022-03-11T19:07:06Z DEBUG unison_fsmonitor] event: Input("")
[2022-03-11T19:07:06Z DEBUG unison_fsmonitor] >> ERROR Unrecognized%20cmd%3A%20
I believe that this is being emitted by the unison-fsmonitor
on my remote host, which is being built from the current tip of your master
branch.
I am not, as far as I can tell, out of space, either locally or remotely.
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 24200302 542 24199760 1% /dev
tmpfs 24202461 2 24202459 1% /dev/shm
tmpfs 24202461 707 24201754 1% /run
tmpfs 24202461 16 24202445 1% /sys/fs/cgroup
/dev/nvme0n1p1 31457280 17799544 13657736 57% /
tmpfs 24202461 5 24202456 1% /run/user/272766
tmpfs 24202461 1 24202460 1% /run/user/0
tmpfs 24202461 1 24202460 1% /run/user/221009
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 93G 0 93G 0% /dev
tmpfs 93G 0 93G 0% /dev/shm
tmpfs 93G 2.1M 93G 1% /run
tmpfs 93G 0 93G 0% /sys/fs/cgroup
/dev/nvme0n1p1 473G 312G 161G 66% /
tmpfs 19G 0 19G 0% /run/user/272766
tmpfs 19G 0 19G 0% /run/user/0
tmpfs 19G 0 19G 0% /run/user/221009
(all of this is in a directory on the /
mount)
I've "worked around" this by switching to a different fsmonitor, which is ... okay, I guess, but yours has been the most consistently stable of the ones I've used so far, and I would prefer to keep using it if I could. I'm not sure how to debug this on the server side, though.
To reproduce (I'm on macOS 12.3.1):
mkdir /tmp/a /tmp/b
unison /tmp/a /tmp/b -repeat watch -ui text -debug fswatch+
While this is running, try
echo $RANDOM > /tmp/a/x
I see a bunch of output, quoted below. Importantly, there is a delay between "Synchronization complete" being printed and new WAIT
commands being sent. If you modify /tmp/a/x
again during this interval, a second synchronization does not occur, and so /tmp/a/x
and /tmp/b/x
stay out of sync (until one of them is changed yet again or Unison is restarted).
I can reproduce this on Unison 2.51.5 as well as the latest master
at time of writing. It doesn't occur when using Unison's built-in fsmonitor implementation on Linux.
The cause appears to be the timing of CHANGES
responses. This fsmonitor implementation sends CHANGES
immediately as soon as changes occur. But the built-in fsmonitor has more of a polling model, where CHANGES
responses are sent only in response to WAIT
commands; a CHANGE
response terminates a wait, and changes that occur while not waiting should be reported at the next wait. In fact, Unison itself seems to ignore the arguments to CHANGES
responses; it simply uses CHANGES
as a notification that it should terminate the wait and send its own CHANGES
command to request filenames of the files that changed. (Nevertheless, the built-in fsmonitor does include the hashes of changed files in its CHANGES
response; there can be multiple hashes in a single response.)
Edit: To further clarify: WAIT
is like select()
, terminated by a CHANGES
response, and CHANGES
commands are like read()
. They affect the same state (list of pending changes). I think.
[fswatch+] >> CHANGES d03c239c5bae8b7a9a35a6284d8a9512
Looking for changes
[fswatch+] << CHANGES d03c239c5bae8b7a9a35a6284d8a9512
[fswatch+] >> RECURSIVE x
[fswatch+] >> DONE
[fswatch+] << START d03c239c5bae8b7a9a35a6284d8a9512 /private/tmp/a x
[fswatch+] >> OK
[fswatch+] << DONE
[fswatch+] << CHANGES ee6f81089ac5b4e616740c34d688212d
[fswatch+] >> RECURSIVE x
[fswatch+] >> RECURSIVE %2Eunison%2Ex%2Eee6f81089ac5b4e616740c34d688212d%2Eunison%2Etmp
[fswatch+] >> DONE
[fswatch+] << START ee6f81089ac5b4e616740c34d688212d /private/tmp/b x
[fswatch+] >> OK
[fswatch+] << DONE
Reconciling changes
changed ----> x
a : changed file modified on 2022-05-09 at 19:05:43 size 6 rw-r--r--
b : unchanged file modified on 2022-05-09 at 19:05:39 size 6 rw-r--r--
Propagating updates
Unison 2.51.5 (ocaml 4.12.1) started propagating changes at 19:05:44.67 on 09 May 2022
[BGN] Updating file x from /private/tmp/a to /private/tmp/b
[END] Updating file x
Unison 2.51.5 (ocaml 4.12.1) finished propagating changes at 19:05:44.67 on 09 May 2022, 0.001 s
Saving synchronizer state
Synchronization complete at 19:05:44 (1 item transferred, 0 skipped, 0 failed)
[fswatch+] >> CHANGES ee6f81089ac5b4e616740c34d688212d
[fswatch+] >> CHANGES ee6f81089ac5b4e616740c34d688212d
[fswatch+] >> CHANGES ee6f81089ac5b4e616740c34d688212d
[fswatch+] << WAIT d03c239c5bae8b7a9a35a6284d8a9512
[fswatch+] << WAIT ee6f81089ac5b4e616740c34d688212d
This looks like a great tool, and fills an important gap in Unison, but it makes it difficult for people to use it if there is no license.
Check out https://choosealicense.com/ for help picking one. Thanks!
My unison-script is crashing with the following output:
Looking for changes
Invalid argument: unquote
Raised at Lwt.poll in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/lwt/lwt.ml", line 126, characters 16-23
Called from Lwt_unix_impl.run in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/lwt/generic/lwt_unix_impl.ml", line 102, characters 8-23
Called from Uitext.Body.synchronizeOnce in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/uitext.ml", line 1124, characters 16-56
Called from Uitext.Body.synchronizePathsFromFilesystemWatcher.loop in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/uitext.ml", line 1183, characters 6-125
Called from Uitext.Body.synchronizeUntilDone in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/uitext.ml", line 1231, characters 8-47
Called from Uitext.Body.start in file "/private/tmp/unison-20211220-29288-1iikm63/unison-2.51.5/src/uitext.ml", line 1417, characters 21-43
More info and a case of how to reproduce can be found at the unison repo: bcpierce00/unison#338 (comment)
According to the maintainers there:
The '%' character is not percent-encoded and this is what's causing the error.
I misread the source. Unison does encode the percent character.
This is a bug in the external fsmonitor.
I'm not that familiar with Rust ... if you would have time to confirm the issue and fix it, I would really appreciate it!
I'm trying to setup unison
to sync a folder between local linux and remote macOS. I installed your utility on macOS, and get this error when running unison
from linux with repeat = watch
:
Fatal error: Server: Error in sending command to filesystem watcher:
Broken pipe [write()]
Do you know what is causing this?
unison: 2.51.2
macOS: 10.12.6
unison -version
unison version 2.51.2 (ocaml 4.08.1)
unison config
root = /Volumes/DevDesktop
root = ssh://[email protected]//workplace/mbazzan
ignore = Name .FBCIndex
ignore = Name .FBCLockFolder
ignore = Name .fseventsd
ignore = Name {.Trash*,.VolumeIcon.icns,.HSicon,TemporaryItems,.Temporary*,TheFindByContentFolder}
ignore = Name .DocumentRevisions-*
ignore = Name {TheVolumeSettingsFolder,.Metadata,.filler,.idsff,.Spotlight,.DS_Store,.CFUserTextEncoding}
ignore = Name */src/*/eclipse-bin
ignore = Name */AAA/*
ignore = Name */.project
ignore = Name */.settings
ignore = Name */.classpath
ignore = Name *.swp
ignore = Name */env
ignore = Name */RemoteSystemsTempFiles
ignore = Name */.idea
ignore = Name */src/*/*.iml
ignore = Name export
ignore = Path moonfish
ignore = Path */build/*/*/*/*/build/{,.}?*
ignorenot = Path */build/*/*/*/*/build/unit-tests
ignore = Regex .*/runtime/perl/lib/perl...-dist/[pP]od
repeat = 5
maxbackups = 0
retry = 1
auto = true
batch = true
confirmbigdeletes = true
times = true
prefer = newer
terse = true
contactquietly = true
log = true
logfile = /Users/mbazzan/.unison/unison.log
ignorecase = false
fastcheck = true
sshargs = -C
command:
RUST_LOG=debug unison -repeat watch
output
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: event: Input("VERSION 1\n")
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: >> VERSION 1
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: event: Input("START 2fa3d0d616137cebf6df2910a9a50132 /Volumes/DevDesktop \n")
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: replicas: {"2fa3d0d616137cebf6df2910a9a50132": Replica { root: "/Volumes/DevDesktop", paths: {"/Volumes/DevDesktop"}, pending_changes: {} }}
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: >> OK
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: event: Input("DIR \n")
DEBUG 2020-06-24T09:12:55Z: unison_fsmonitor: >> OK
...
...
...
...
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
[END] Copying BluTraceBuilderTools/build
[END] Copying BluTraceBuilderTools/logs
[END] Updating file BluTraceBuilderTools/packageInfo
[END] Copying BluTraceBuilderTools/src/BluTraceBuilderTools/.coverage
[END] Copying BluTraceBuilderTools/src/BluTraceBuilderTools/.pytest_cache
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/doc/.unison._apidoc.2fa3d0d616137cebf6df2910a9a50132.unison.tmp"), op: Ok(CHMOD | CREATE | RENAME), cookie: Some(89893770) })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/doc/_apidoc"), op: Ok(RENAME), cookie: Some(89893770) })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/src/blu_trace_builder_tools/.unison.extracommands.py.2fa3d0d616137cebf6df2910a9a50132.unison.tmp"), op: Ok(CHMOD | CREATE | RENAME | WRITE), cookie: None })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/src/blu_trace_builder_tools/extracommands.py"), op: Ok(RENAME), cookie: None })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/test_profile/fsts/development/fst_data/Video_BrowseVideoIntent/.unison..build.manifest.2fa3d0d616137cebf6df2910a9a50132.unison.tmp"), op: Ok(CHMOD | CREATE | RENAME | WRITE), cookie: Some(89893779) })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: event: FSEvent(RawEvent { path: Some("/Volumes/DevDesktop/BluTraceBuilderTools/src/BluTraceBuilderTools/test_profile/fsts/development/fst_data/Video_BrowseVideoIntent/.build.manifest"), op: Ok(RENAME), cookie: Some(89893779) })
DEBUG 2020-06-24T09:13:25Z: unison_fsmonitor: >> CHANGES 2fa3d0d616137cebf6df2910a9a50132
[END] Copying BluTraceBuilderTools/src/BluTraceBuilderTools/doc/_apidoc
[END] Updating file BluTraceBuilderTools/src/BluTraceBuilderTools/src/blu_trace_builder_tools/extracommands.py
[END] Copying BluTraceBuilderTools/src/BluTraceBuilderTools/test_profile/fsts/development/fst_data/Video_BrowseVideoIntent/.build.manifest
Synchronization complete at 11:13:26 (12 items transferred, 0 skipped, 0 failed)
DEBUG 2020-06-24T09:13:27Z: unison_fsmonitor: event: Input("WAIT 2fa3d0d616137cebf6df2910a9a50132\n")
Fatal error: Server: No file monitoring helper program found
DEBUG 2020-06-24T09:13:27Z: unison_fsmonitor: event: Input("")
DEBUG 2020-06-24T09:13:27Z: unison_fsmonitor: >> ERROR Unrecognized cmd:
and it exits
There have been two pretty important bug fixes in the past two weeks[0][1]. This package wasn't usable for me before these fixes and I presume for lots of other people. I was able to build locally and it worked great. Any chance of having new packages published as a new patch version?
Thank you for the great contribution!
Hello! We're successfully using unison in fairly high volume with a macOS M1 binary we've compiled. This is a request to see if you can compile your rust project on an M1 and perhaps update the homebrew release.
If you don't want to, it would be nice to get a copy of the entire rust project so we can give it a go.
Big thanks :)
especially when running Unison, I quite often experienced the following error:
Fatal error: exception Unix.Unix_error(Unix.EBADF, "set_nonblock", "")
Running "strace unison-fsmonitor" yielded:
fcntl(0, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(0, F_SETFL, O_RDWR|O_NONBLOCK) = 0
fcntl(1, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(1, F_SETFL, O_RDWR|O_NONBLOCK) = 0
inotify_init() = -1 EMFILE (Too many open files)
fcntl(-1, F_GETFL) = -1 EBADF (Bad file descriptor)
write(2, "Fatal error: exception Unix.Unix"..., 71Fatal error: exception Unix.Unix_error(Unix.EBADF, "set_nonblock", "")) = 71
exit_group(2) = ?
+++ exited with 2 +++
I have fiddled around with 'ulimit -n', without lasting success.
The real solution was to change the following:
sysctl -w fs.inotify.max_user_instances=256 >> /etc/sysctl.d/unison.conf
The original OS vendor setting (OpenSuSE) was at 128. And with Nextcloud and another sync tool running besides Unison my system seemed to have to many inotify resources used.
I would appreciate if you could add a hint to your documentation somewhere, that inotify limits may have to be increased if the before mentioned error is encountered. The issue may be related to #8 .
Running:
brew update
brew upgrade
brew doctor <- all good
brew install autozimu/homebrew-formulas/unison-fsmonitor
I got:
Updating Homebrew...
==> Tapping autozimu/formulas
Cloning into '/usr/local/Homebrew/Library/Taps/autozimu/homebrew-formulas'...
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 16 (delta 1), reused 8 (delta 1), pack-reused 8
Unpacking objects: 100% (16/16), done.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/autozimu/homebrew-formulas/Formula/unison-fsmonitor.rb
invalid attribute for formula 'autozimu/formulas/unison-fsmonitor': version (nil)
Error: Cannot tap autozimu/formulas: invalid syntax in tap!
Hi,
I'm seeing following error
Fatal error: Server: Filesystem watcher error: cannot add a watcher: system limit reached
The watcher can be disabled by setting preference 'watch' to false
Any suggestion please? Thank you.
Similar to #12
When adding the formula and installing the package, the system receives the MacOS binary that of course cannot be executed.
$ file /home/linuxbrew/.linuxbrew/Cellar/unison-fsmonitor/0.3.0/bin/unison-fsmonitor
/home/linuxbrew/.linuxbrew/Cellar/unison-fsmonitor/0.3.0/bin/unison-fsmonitor: Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE|HAS_TLV_DESCRIPTORS>
Do you have any plan to support different platforms/architecture?
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.