dmatveev / libinotify-kqueue Goto Github PK

View Code? Open in Web Editor NEW

34.0 34.0 7.0 516 KB

NetBSD Google Summer of Code 2011 project (#2)

License: Other

Shell 0.01% C 55.44% C++ 37.15% Makefile 0.64% M4 6.74%

libinotify-kqueue's People

Contributors

Stargazers

Watchers

Forkers

stass karlpilkington zhanglei mephi-ut slowfranklin 5nake hazzasym

libinotify-kqueue's Issues

#include <sys/tree.h> is missing on FreeBSD

Lost deletion events

I use the current version of code from in_isdir branch on FreeBSD 10.0-RELEASE #0 r260789 amd64 with simple demo software, described below. I found, that libinotify-kqueue works properly during some short time and fails to detect deletion events after that. Even after reboot. I'm not sure, but this problem may be related with pipe creation in watching directory. But this pipe was deleted before next booting. Source code of used tool is based on http://man7.org/tlpi/code/online/dist/inotify/demo_inotify.c.html

#include <sys/inotify.h>
#include <stdio.h>
#include <unistd.h>

#define BUF_LEN (1<<16)

int main(int argc, char *argv[])
{
        int fd, wd, numRead;
        char buf[BUF_LEN];

        fd = inotify_init();
        wd = inotify_add_watch(fd, argv[1], IN_ALL_EVENTS);

        for (;;)
        {
                numRead = read(fd, buf, BUF_LEN);
                if (numRead > 0)
                {
                        printf("Got something\n");
                }
        }

        return 0;
}

P.S. Sorry for my English

use-after-free in worker_sets_extend/worker_update_flags

 78 worker_sets_extend (worker_sets *ws,
...
 86         void *ptr = NULL;
 87         ptr = realloc (ws->events, sizeof (struct kevent) * to_allocate);
 88         if (ptr == NULL) {

realloc can move ws->events to a different memory location, thus invalidating all pointers in ws->watches[...]->event.

These pointers are then dereferenced in worker_update_flags:

410 worker_update_flags (worker *wrk, watch *w, uint32_t flags)
...
415     w->flags = flags;
416     w->event->fflags = inotify_to_kqueue (flags, w->is_really_dir, 0);
417 
...
428                 depw->flags = flags;
429                 depw->event->fflags = inotify_to_kqueue (flags, ...
430             }

w->event can here point to freed memory, or into some other data structure.

In this core dump I found, they pointed into the ws->watches array after reallocation. The upper 32 bits of some pointers have been corrupted when w->event->fflags has been set by following an invalid pointer.

(gdb) x/32xg $rcx
0x81149ac00:    0x0000000000000000      0x0000000811418280
0x81149ac10:    0x0000000811418490      0x00000008114184c0
0x81149ac20:    0x00000008114184f0      0x0000001e11418520 <--
0x81149ac30:    0x0000000811418550      0x0000000811418580
0x81149ac40:    0x00000008114185b0      0x0000001e114185e0 <--
0x81149ac50:    0x0000000811418610      0x0000000811418640
0x81149ac60:    0x0000000811418670      0x0000001e114186a0 <--
0x81149ac70:    0x00000008114186d0      0x0000000811418700
0x81149ac80:    0x00000008114187f0      0x0000001e114187c0 <--
0x81149ac90:    0x0000000811418820      0x00000008114188b0
0x81149aca0:    0x00000008114188e0      0x0000000811418880
0x81149acb0:    0x0000000811418940      0x00000008114189a0
0x81149acc0:    0x00000008114189d0      0x0000000811418a00
0x81149acd0:    0x0000000811418a30      0x0000000811418a60
0x81149ace0:    0x0000000811418a90      0x0000000811418ac0
0x81149acf0:    0x0000000811418b20      0x0000000811418b50

The corrupted pointers caused a subsequent crash in worker_add_or_modify.

Question: Only watch directories

I maintain forked-daapd which uses inotify for Linux and kqueue for FreeBSD to watch the users music library. For FreeBSD, I would like to switch to libinotify-kqueue, since that would make maintenance much easier. However, music libraries can be quite large, so I can't let libinotify-kqueue watch every file, since that will lead to too many open files.

To get around this, the current kqueue solution in forked-daapd only opens/watches directories. So I was wondering if there is some way to make libinotify also only open directories?

incrond fails at monitoring directories

Hi,

On FreeBSD I built incron (on top of your libinotify) :
http://inotify.aiken.cz/?section=incron&page=about&lang=en

it works great to minitor files but it fails at monitoring directories:
$ ./incrontab -l
/tmp IN_ALL_EVENTS logger toto

Nothing happen when I modify files in /tmp

[suggestion] Implement command line interface like 'inotifywait' in linux

Currently there is no way to wait for file deletion event (for example) in the shell program. So 'inotifywait' would be very useful, and should also be very easy to add having the library ready. Also here is the most natural place for it. Your project will become so much more useful.

"make test" sometimes gets killed with SIGPIPE

A rare but quite reproducible issue (at least) on NetBSD 6.99.

The deletion events are not provided in time

 $ gcc test.c -L.libs -I. -linotify -o test
 $ test /tmp/test

 $ touch /tmp/test/1
 $ touch /tmp/test/2
 $ rm /tmp/test/1
 $ rm /tmp/test/2

The IN_DELETE notification for 2 will be sent only when something else will occur in the directory.

Doesn't set IN_ISDIR if new directory is created

Hello.

I'm continuing to port my "clsync" to FreeBSD using libinotify and found another problem.

When new directory is created "event->mask == 0x0100" (using libinotify), but should be "event->mask == Event 0x40000100" (as using native inotify).

Test results: FreeBSD-10.2

Hi,

I'm getting the following on FreeBSD-10.2 using version b2873cf:

..........x............x.....x......x....x.......xx......................................x...x....x.x.....x.x..........

In test "Directory notifications":
    failed: receive IN_MOVED_FROM event on moving file from directory to another location within the same mount point
    failed: receive IN_MOVED_TO event on moving file to directory from another location within the same mount point

In test "Open/close notifications":
    failed: receive IN_OPEN on cat
    failed: receive IN_CLOSE_NOWRITE on cat
    failed: receive IN_OPEN on ls
    failed: receive IN_CLOSE_NOWRITE on ls
    failed: receive IN_OPEN on modify
    failed: receive IN_CLOSE_WRITE on modify

In test "Symbolic links":
    failed: Start watch successfully on a symlink file with IN_DONT_FOLLOW
    failed: Receive IN_ATTRIB after touching symlink itself
    failed: Receive IN_MOVE_SELF after moving the symlink
    failed: Receive IN_DELETE_SELF after removing the symlink

In test "Failures":
    failed: watch id is -1, errno set to EACCES when starting watching a file without read access

--------------------
     Run: 119
  Passed: 106
  Failed: 13

# uname -a 
FreeBSD stingray.adestra.com 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015     [email protected]:/usr/obj/usr/src/sys/GENERIC  amd64

Configure command line:

$ ./configure --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/man --infodir=/usr/local/info/ --build=amd64-portbld-freebsd10.2

Aside from the expected IN_OPEN, IN_CLOSE_WRITE and IN_CLOSE_NOWRITE failures what am I missing here to get all the tests passing?

Mistakenly close down fd 0 upon watch failure

A problem bothered me a great deal with my samba4 build on FBSD R10, and finally I got some time today to track this problem to a bug in libinotify-kqueue. The version I have is from 20110829, but the bug seems still present in the latest revision in github. Here's a brief description of the problem: a request comes in to watch everything in a directory, not every file in this directory is readable though, so watch_init() would fail for some of them, and leave watch::fd to its initial value 0, the subsequent watch_free() call would then incorrectly close down 0. This mistake would eventually cause kevent() in watch_thread to fail with EBADF and result in an infinite loop. The fix would be very straightforward, initialize watch::fd to -1 instead.

Crashes reported for 271ef97

The following program crashes with the recent changes in libinotify, though it works fine on GNU/Linux:

/* compile: "cc inotify.c -o inotify -linotify -I/usr/local/include -L/usr/local/lib" */
/* run: "./inotify ." */

#include <sys/inotify.h>
#include <limits.h>
#include <err.h>
#include <stdio.h>
#include <sys/types.h>  /* Type definitions used by many programs */
#include <stdio.h>      /* Standard I/O functions */
#include <stdlib.h>     /* Prototypes of commonly used library functions,
               plus EXIT_SUCCESS and EXIT_FAILURE constants */
#include <unistd.h>     /* Prototypes for many system calls */
#include <errno.h>      /* Declares errno and defines error constants */
#include <string.h>     /* Commonly used string-handling functions */

static void             /* Display information from inotify_event structure */
displayInotifyEvent(struct inotify_event *i)
{
    printf("    wd =%2d; ", i->wd);
    if (i->cookie > 0)
        printf("cookie =%4d; ", i->cookie);

    printf("mask = ");
    if (i->mask & IN_ACCESS)        printf("IN_ACCESS ");
    if (i->mask & IN_ATTRIB)        printf("IN_ATTRIB ");
    if (i->mask & IN_CLOSE_NOWRITE) printf("IN_CLOSE_NOWRITE ");
    if (i->mask & IN_CLOSE_WRITE)   printf("IN_CLOSE_WRITE ");
    if (i->mask & IN_CREATE)        printf("IN_CREATE ");
    if (i->mask & IN_DELETE)        printf("IN_DELETE ");
    if (i->mask & IN_DELETE_SELF)   printf("IN_DELETE_SELF ");
    if (i->mask & IN_IGNORED)       printf("IN_IGNORED ");
    if (i->mask & IN_ISDIR)         printf("IN_ISDIR ");
    if (i->mask & IN_MODIFY)        printf("IN_MODIFY ");
    if (i->mask & IN_MOVE_SELF)     printf("IN_MOVE_SELF ");
    if (i->mask & IN_MOVED_FROM)    printf("IN_MOVED_FROM ");
    if (i->mask & IN_MOVED_TO)      printf("IN_MOVED_TO ");
    if (i->mask & IN_OPEN)          printf("IN_OPEN ");
    if (i->mask & IN_Q_OVERFLOW)    printf("IN_Q_OVERFLOW ");
    if (i->mask & IN_UNMOUNT)       printf("IN_UNMOUNT ");
    printf("\n");

    if (i->len > 0)
        printf("        name = %s\n", i->name);
}

#define BUF_LEN (10 * (sizeof(struct inotify_event) + NAME_MAX + 1))

int
main(int argc, char *argv[])
{
    int inotifyFd, wd, j;
    char buf[BUF_LEN] __attribute__ ((aligned(8)));
    ssize_t numRead;
    char *p;
    struct inotify_event *event;

    if (argc < 2 || strcmp(argv[1], "--help") == 0)
        printf("%s pathname...\n", argv[0]);

    inotifyFd = inotify_init();                 /* Create inotify instance */
    if (inotifyFd == -1)
        perror("inotify_init");

   /* For each command-line argument, add a watch for all events */

    for (j = 1; j < argc; j++) {
        wd = inotify_add_watch(inotifyFd, argv[j], (IN_MODIFY | IN_ATTRIB |
            IN_MOVE | IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_MOVE_SELF));
        if (wd == -1)
            perror("inotify_add_watch");

        printf("Watching %s using wd %d\n", argv[j], wd);
    }

    inotify_rm_watch(inotifyFd, wd);

    for (;;) {                                  /* Read events forever */
        numRead = read(inotifyFd, buf, BUF_LEN);
        if (numRead == 0)
            perror("read() from inotify fd returned 0!");

        if (numRead == -1)
            perror("read");

        printf("Read %ld bytes from inotify fd\n", (long) numRead);

        /* Process all of the events in buffer returned by read() */

        for (p = buf; p < buf + numRead; ) {
            event = (struct inotify_event *) p;
            displayInotifyEvent(event);

            p += sizeof(struct inotify_event) + event->len;
        }
    }

    exit(EXIT_SUCCESS);
}

Hanging while inotify_add_watch(): race conditions

Hello. I'm porting my utility clsync to FreeBSD with using of libinotify.

First of all huge thanks for the project!

However inotify_add_watch() hangs in my case while calling it this way:

inotify_add_watch(4, "/root/clsync/examples/testdir/from/1", 0x2000fce)

Directory /root/clsync/examples/testdir/from/1 contains only directories 2 and 2/3.

Here's the backtrace:

(gdb) info threads
  4 Thread 802806400 (LWP 101152/clsync)  0x00000008016c889c in __error () from /lib/libthr.so.3
  3 Thread 802808400 (LWP 100150/clsync)  0x00000008014b1647 in ik_barrier_impl_wait (impl=0x802859160) at /usr/ports/devel/libinotify/work/dmatveev-libinotify-kqueue-2a0d601/barriers.c:59
* 2 Thread 802808000 (LWP 100149/clsync)  0x000000080194362a in _sigwait () from /lib/libc.so.7
(gdb) thread 2
[Switching to thread 2 (Thread 802808000 (LWP 100149/clsync))]#0  0x000000080194362a in _sigwait () from /lib/libc.so.7
(gdb) bt
#0  0x000000080194362a in _sigwait () from /lib/libc.so.7
#1  0x00000008016c224a in sigwait () from /lib/libthr.so.3
#2  0x000000000040d0c5 in ?? ()
#3  0x0000000802808000 in ?? ()
#4  0x0000000802808008 in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x00007fffffbfdff0 in ?? ()
#7  0x0000000000000000 in ?? ()
(gdb) thread 3
[Switching to thread 3 (Thread 802808400 (LWP 100150/clsync))]#0  0x00000008014b1647 in ik_barrier_impl_wait (impl=0x802859160)
    at /usr/ports/devel/libinotify/work/dmatveev-libinotify-kqueue-2a0d601/barriers.c:59
59          while (impl->sleeping != 0);
(gdb) bt
#0  0x00000008014b1647 in ik_barrier_impl_wait (impl=0x802859160) at /usr/ports/devel/libinotify/work/dmatveev-libinotify-kqueue-2a0d601/barriers.c:59
#1  0x00000008014b1535 in ik_barrier_wait (b=0x802859160) at /usr/ports/devel/libinotify/work/dmatveev-libinotify-kqueue-2a0d601/barriers.c:118
#2  0x00000008014af07b in process_command (wrk=0x802859100) at /usr/ports/devel/libinotify/work/dmatveev-libinotify-kqueue-2a0d601/worker-thread.c:110
#3  0x00000008014af5bd in worker_thread (arg=0x802859100) at /usr/ports/devel/libinotify/work/dmatveev-libinotify-kqueue-2a0d601/worker-thread.c:607
#4  0x00000008016bd4a4 in pthread_create () from /lib/libthr.so.3
#5  0x0000000000000000 in ?? ()
(gdb) thread 4
[Switching to thread 4 (Thread 802806400 (LWP 101152/clsync))]#0  0x00000008016c889c in __error () from /lib/libthr.so.3
(gdb) bt
#0  0x00000008016c889c in __error () from /lib/libthr.so.3
#1  0x00000008016c6d5c in _pthread_cond_wait () from /lib/libthr.so.3
#2  0x00000008014b1609 in ik_barrier_impl_wait (impl=0x802859160) at /usr/ports/devel/libinotify/work/dmatveev-libinotify-kqueue-2a0d601/barriers.c:51
#3  0x00000008014b1535 in ik_barrier_wait (b=0x802859160) at /usr/ports/devel/libinotify/work/dmatveev-libinotify-kqueue-2a0d601/barriers.c:118
#4  0x00000008014ae6b5 in worker_cmd_wait () from /usr/local/lib/libinotify.so.0
#5  0x00000008014ae3ca in inotify_add_watch () from /usr/local/lib/libinotify.so.0
#6  0x0000000000409a73 in ?? ()
#7  0x0000000000000000 in ?? ()

CPU is highly loaded while the problem:

CPU: 96.9% user,  0.0% nice,  1.2% system,  1.9% interrupt,  0.0% idle
93628 root          3 101    0 33996K  3796K RUN      0:44 100.00% clsync

Seems to be a problem with pthread-conflict between clsync and libinotify.

Propagate changes from #12 to gio-kqueue

Unfortunately, the Glib GIO still uses its own copy of dep-list.[ch]. Recently some changes have been made (#12), and these changes should be reflected in the Glib's main tree.

Opening a ticket here just to keep it in mind.

wulf7/libinotify-kqueue integration

Hi, Dmitry
I had almost completed refactoring of my work and placed it into integrate-dmatveev branch of wulf7/libinotify-kqueue repo
All changes can be combined into following parts:
5eae094 - 2e370b2 move barriers code to compat.c
b3e9304 - 147e07d remove some watch deletion optimizations for code simplification
a40e2f7 - 6c30885 Combine single and bulk write code paths using writev scatter-gather operations
632734b - dab864e Replace linked list implementation with BSD queue(3) macroses and make some directory diff optimizations
182cfa0 - 497b349 Switch to POSIX.1-2008 relative pathname functions for file operations.
ae82cd9 - f1624ee open() flags improvements
8e1afe4 - 09aca85 Make most of internal processing inotify-watch centric. E.g. split worker_sets on per inotify_watch basis
228e972 - 3f70da7 Use inode numbers instead filenames for processing of kqueue watches (like kernel does)
6e28faf - bfb6346 Replace worker-set arrays with more flexible RB-tree based structures
5204da0 - 32da88e Implement most of inotify_add_or_modify flags
e4d0d92 - 99aeec4 Some fixes and optimizations of kqueue watch creation
4e4707a - a242780 Miscellaneous small fixes
3f71d2e - 3ee44d5 Implementation of events which require kernel patching (IN_OPEN, IN_CLOSE, IN_ACCESS)
340089c - ... etc

Resulting code is not extensively tested but is able to pass test suite under valgrind control with jemalloc memory debugging features enabled.

Build failure: Missing file barriers.c

-deleted-

Where is the documentation?

Hello.

Is there anywhere a documentation about how to use the library?

libinotify open() considerations

libinotify tries to open named pipes (FIFOs) as well as other files. And this leads to lockups because those files are usually either already open, or would be open soon. In particular, that breaks Kile 2.1 (KDE4 version) on OpenBSD, when "inotify" method is used in KDirWatch: Kile creates named pipes in home directory to talk with LyX server, and then tries to use them. But there is a KDirWatch object being created at other place in program, that monitors home directory, too. As a result, Kile gets stuck at watch_init().

Even more, actually Kile creates symlinks to pipes created under /tmp; this uncovers that libinotify doesn't use O_NOFOLLOW when adding a watch - IMHO, that's a bug, too.