e-dant / watcher Goto Github PK
View Code? Open in Web Editor NEWFilesystem watcher. Works anywhere. Simple, efficient and friendly.
License: MIT License
Filesystem watcher. Works anywhere. Simple, efficient and friendly.
License: MIT License
Decide which one you want:
README.md
and .tellfile
into include/watcher
include/watcher/*
up one levelThank you for the library. I have a one comment:
static
as you cannot create multiple watchers (the fact that it is an inline function does not change the nature of the static
variable)https://github.com/e-dant/watcher/blob/release/sinclude/watcher/watcher.hpp#L838
static auto callback_hook = callback; // here
const auto callback_adapter =
[](ConstFSEventStreamRef, /* stream_ref
(required) */
auto*, /* callback_info (required) */
You can simply pass the hook in the lambda:
const auto callback_adapter =
[&callback_hook = callback](ConstFSEventStreamRef, /* stream_ref
(required) */
auto*, /* callback_info (required) */
If anyone find themselves in a spot where they need to create compile definitions or patches to get this project working the way it should, please drop a comment here. I'm happy to address the issues.
Some possible requests you can make:
Create version definitions within the project.
Create a basic test program (without running the full test suite) to see if the project compiles and/or runs on some platform.
Fix X thing on Y compiler.
Etc, etc.
Whatever you need.
I once rewrote this project in Rust to explore a safer language. Because this project is a thin, uniform wrapper around other unsafe systems, the rewrite wasn't safer.
It did turn out to be, in this project's case, a much more usable, ergonomic and mature ecosystem for
All of which are things I want to work on more.
A Rust wrapper, without changing our underlying implementation, might do the trick.
Leave comments here for test cases.
I'm looking for interesting ways to attempt to break the watchers.
Our current test suite creates many thousands of files and/or directories. This is done manually while waiting for an error.
The test suite should be expanded and automated.
Recently, on both the release
and next
branches, our CodeQL builds have been coming up with this error:
https://github.com/e-dant/watcher/actions/runs/6684346814/job/18161503474#step:5:74
We also found this error on a Clang build for Linux in development:
https://github.com/e-dant/watcher/actions/runs/6684014904/job/18160816032#step:4:58
This looks and feels like a kind of issue unrelated to our codebase. I am not sure why the issue has only just started to crop up.
Using Nix is always an option to make our builds reproducible and mostly deterministic. Nix was mentioned in a previous build issue: #33
There are some problems with the sanitizers on Linux and Windows that we need to fix.
On Linux hosts, the address and memory sanitizers seem to be broken.
The address sanitizer fails here.
Which, as best I can tell, is a bug in LLVM's asan. Others seem to have similar issues, although their workarounds don't work locally, and CI time adds up, so I haven't deployed their workarounds.
The memory sanitizer is also in a bad state, saying that some standard library string functions are being used uninitialized from Catch.
The sanitizers on windows, I'm assuming, have different link options. The documentation is unhelpful because it should work as-is and seems to only support one sanitizer, asan.
The messaging syntax (including errors) should be detailed in the readme.
Messages come through event
. The where
field looks like this:
[s, e]/[sys, self]/[description]
Where s
means "successful" (such as s/self/live@path
) and e
means "error" (such as e/sys/read
).
self
means we failed. sys
means the operating system failed.
The description
is usually a function name, an api, or a path. It's something extra.
These events always have the kind
field as watcher
(as opposed to file
, dir
, etc).
The what
field varies. When a watcher first starts watching, what
is create
. When a watcher dies, it's destroy
. Error messages are usually other
.
We need to put this in the readme.
We need to document what we use from each header and remove unused includes, at least in include/watcher/*
and src/watcher/*
files.
We should investigate boot::asio in these places:
Since 0.8.1, MSVC 19.33 causes compile error.
https://godbolt.org/z/qj6W84KP7
example.cpp
C:\Users\ContainerAdministrator\AppData\Local\Temp\compiler-explorer-compiler2023313-18468-1p7mtrd.iwwy\raw.githubusercontent.com/e-dant/watcher/release/0.8.3/include/wtr/watcher.hpp(191): error C3431: 'kind': a scoped enumeration cannot be redeclared as an unscoped enumeration
C:\Users\ContainerAdministrator\AppData\Local\Temp\compiler-explorer-compiler2023313-18468-1p7mtrd.iwwy\raw.githubusercontent.com/e-dant/watcher/release/0.8.3/include/wtr/watcher.hpp(177): error C3431: 'what': a scoped enumeration cannot be redeclared as an unscoped enumeration
Compiler returned: 2
I do not know the reason for this compile error.
Is there any workaround?
In 0.9.0, watcher uses "alternative operator representations".
For example, following code uses and
operator.
https://github.com/e-dant/watcher/blob/release/0.9.0/include/wtr/watcher.hpp#L111-L112
MSVC doesn't understand "alternatvie operator representations" in default.
With /permissive-
compiler flag, MSVC can understand it.
I think there are two way to solve it.
/permissive-
compiler flag to CMakeLists.txt.I think both are effective.
Can you please decide on a solution?
We need performance benchmarks against the other filesystem watchers. I'm thinking:
We have performance tests, but they're not comparative.
Hi @e-dant, I am a bit unsure as to what event::effect_type
values are set for which filesystem events. This might be due to my lack of experience with filesystem events, or due to my debian bookworm setup (I am using Debian Bookworm with KDE Plasma 5.27.5). Could you clarify a few points, please?
cp new_file watched_dir/
command, 2 events are triggered event::effect_type::create
first and then event::effect_type::modify
. Is this intended or should only 1 event be set?mv old_filename new_filename
command, only the event::effect_type::rename
with the old path is set. How should I get the new filename? Shouldn't a event::effect_type::create
be set?mv new_file watched_dir/
command to move a file into the watched directory does not trigger any events at all. Shouldn't this set event::effect_type::create
event?mv filename ..
in the watched directory triggers a event::effect_type::rename
and not event::effect_type::destroy
. Is this intended?event::effect_type::modify
is set twice after file modifications are saved. Is this intended?.filename.txt.swp
or .filename.txt.kate-swp
for kate) is created. This file also triggers watch events, it's not hard to filter them out by checking the event::path_name
filename content, but shouldn't these files not trigger any watch events?At some point in the last two weeks, it looks like the GitHub hosted Windows runner changed.
For some reason I do not understand, Windows is now failing to execute the test artifacts: https://github.com/e-dant/watcher/actions/runs/6426774299/job/17451441610
It is not clear to me what the error is, or what changed that may have affected this since https://github.com/e-dant/watcher/actions/runs/6140888668
I've read the source code and I was disappointed not to find any usage of kernel's primitive to watch for filesystem changed (except for Darwin).
Why don't you use kernel primitives (like inotify on linux, for example)?
IMHO, it'll be a lot more efficient than computing a giant hash table of path to monitor.
First of all, thank you for this library!
I am seeing something odd that I didn't expect. When watching a directory with a bunch of files, when a load event is occurring on any of those files, a "modify" seems to be sent down in Windows 11. This is surprising, as the file itself isn't modified (though I do correctly see modifies when one occurs!).
Upon further reading online, it seems like the value FILE_NOTIFY_CHANGE_LAST_ACCESS
is changing which is being treated as a modify. Is there anything that can be done? I will see if I can reproduce in a small test to confirm.
Necessary (or, at the least, very helpful) for testing and development are containers suitable for building and running Watcher.
I have some containers laying around for this purpose. They should be made available here.
Looking for thoughts and solutions.
This project runs on iOS. I want to integrate iOS into our CI pipeline.
The only method I'm currently aware of is attaching a certificate and provisioning profile to this repository's secrets.
The only such secrets I have are tied to me personally.
That would be fine for now.
The problem I see is complications for future contributors access rights to this repository.
The readme offers Watcher's design goals, sure, but it doesn't offer the non-goals.
We should add a section on the things that Watcher is not interested in solving (at least at the moment). These are the OS-level caveats to certain system APIs. For inotify
, that's events potentially disappearing or being duplicated. For std::filesystem
, there are some places where UB might happen, although I haven't looked into exactly when they occur.
These are important caveats that deserve a place in the readme but require more research.
This library has become somewhat popular. I expect at least one other user is actively using this project. For them/those users, we need to publish upcoming API changes.
They are not fully fleshed out yet, however, at least these two things may be affected:
watch
for delay_ms
may be removed.event.where
will be changed to a tuple or another field will be added to the event object.This will happen before the 1.0 release.
One of the only filesystems we can't watch is the virtual /proc
filesystem on Linux.
There is a way to create an ebpf program for the kernel to run which lets us know about modification events on those paths.
Every so often, we get an error like this: https://github.com/e-dant/watcher/actions/runs/5920128540/job/16051048104#step:5:33
I tried to address this in 38d93a4 by adding a GitHub action to install cmake. Maybe that cmake action isn't working, or maybe something else is off. The path which make is looking for cmake in hints that the brew version of cmake isn't always there. I'm not sure why make keeps looking there.
Maybe the CMake cache needs to be reset? Maybe we should use Nix and just not worry about these kinds of issues?
I'm on Windows 11 and it seems like rename events are sending two events, unlike the example in the README where there is one event with both the old and the new name. First the old name is sent, then the new name and associated
is always nullptr
.
There is only one function that matters (and only one function that should be accessible) to the user: water::watcher::run
.
The header include/watcher/event.hpp
may have some types worth exposing as well.
The only items in namespace literal
should be the ones mentioned above.
I wanted to use this project as a header-only library via conan-center, but found out that the minimum C++ standard is set to C++20 and not C++17.
in v0.2.2 changelog, it is stated that:
C++20 Concepts are removed for being very interesting but not beneficial enough to an API with a very small surface.
However, in e28bd3c commit wtr.watcher.cmake the minimum standard was changed from 17 to 20. Is this only for the standalone program or is it also important for the header-only library files?
Hi @e-dant, thanks for your work on issue #32, however, pure C++17 support is still not available.
wtr/watcher.hpp includes the bit library header, which was introduced in C++20 and is preventing compilation on older systems (for example Debian buster).
Is this header needed?
I am excited with the 0.5.3 version upgrade that wraps not only inotify but also fanotify.
On the other hand, fanotify is a relatively new feature that will not compile on slightly older kernels.
The 0.5.3 implementation of fanotify uses FAN_REPORT_DFID_NAME, so it seems to require Kernel 5.9 or later to compile.
https://man7.org/linux/man-pages/man2/fanotify_init.2.html
Is it possible to add the following condition for include/watcher/adapter/linux/watch.hpp and include/watcher/adapter/linux/fanotify/watch.hpp?
There are some very high-level languages with ecosystems which could benefit from this filesystem watcher.
This is an efficient, reasonably well tested filesystem watcher with a small, relatively stable API.
There isn't much to expose or package.
I think this is mostly a matter of figuring out the "right way" to expose a library with a C ABI to these languages, and then just doing it.
"watch
will happily keep continue watching"... there was an attempt to use words, but it looks like they got mangled like symbols in a binary.
On Apple platforms, the user needs to link against two system libraries.
On most (all?) platforms, c++
should point to the platform's default C++ compiler.
We should change g++
to c++
and put something like $(uname | grep darwin && echo '-framework CoreFoundation -framework CoreServices')
in the compile command from "Quick Start".
The file path in the watch event is coming out to be null for certain file operations on macOS.
The API used in the code https://github.com/e-dant/watcher/blob/release/devel/include/detail/wtr/watcher/adapter/darwin/watch.hpp#L114 is CFStringGetCStringPtr
The official documentation clearly states that this method may return null if the internal implementation in macOS cannot provide a no allocation pointer and should not be relied on.
https://developer.apple.com/documentation/corefoundation/1542133-cfstringgetcstringptr?language=objc
Alternate function that works for me:
inline auto path_from_event_at(void* event_recv_paths, unsigned long i)
-> std::filesystem::path
{
if (event_recv_paths)
if (
void const* from_arr = CFArrayGetValueAtIndex(
static_cast<CFArrayRef>(event_recv_paths),
static_cast<CFIndex>(i)))
if (
void const* from_dict = CFDictionaryGetValue(
static_cast<CFDictionaryRef>(from_arr),
kFSEventStreamEventExtendedDataPathKey)) {
auto cfString = static_cast<CFStringRef>(from_dict);
CFIndex length = CFStringGetLength(cfString);
CFIndex maxSize = CFStringGetMaximumSizeForEncoding(length, kCFStringEncodingUTF8);
char *buffer = new char[static_cast<size_t>(maxSize)];
if (CFStringGetCString(cfString, buffer, maxSize, kCFStringEncodingUTF8)) {
std::string result(buffer);
delete[] buffer;
return result;
} else {
delete[] buffer;
return std::string(); // Return an empty string if conversion fails
}
}
return {};
}
All directories should have, at least, a README.md
and a .tellfile
. Even if they're short.
I need to create docs about contributions. Here's the flow we should generally use:
Push to next
-> await ci/GitHub actions -> run tool/hone
for the single header -> run tool/release
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.