Giter Club home page Giter Club logo

pyinotify's Introduction

Build_Status Coverage_Status

Overview

inotify functionality is available from the Linux kernel and allows you to register one or more directories for watching, and to simply block and wait for notification events. This is obviously far more efficient than polling one or more directories to determine if anything has changed. This is available in the Linux kernel as of version 2.6 .

We've designed this library to act as a generator. All you have to do is loop, and you'll see one event at a time and block in-between. After each cycle (all notified events were processed, or no events were received), you'll get a None. You may use this as an opportunity to perform other tasks, if your application is being primarily driven by inotify events. By default, we'll only block for one-second on queries to the kernel. This may be set to something else by passing a seconds-value into the constructor as block_duration_s.

This project is unrelated to the *PyInotify* project that existed prior to this one (this project began in 2015). That project is defunct and no longer available.

Installing

Install via pip:

$ sudo pip install inotify

Example

Code for monitoring a simple, flat path (see "Recursive Watching" for watching a hierarchical structure):

import inotify.adapters

def _main():
    i = inotify.adapters.Inotify()

    i.add_watch('/tmp')

    with open('/tmp/test_file', 'w'):
        pass

    for event in i.event_gen(yield_nones=False):
        (_, type_names, path, filename) = event

        print("PATH=[{}] FILENAME=[{}] EVENT_TYPES={}".format(
              path, filename, type_names))

if __name__ == '__main__':
    _main()

Output:

PATH=[/tmp] FILENAME=[test_file] EVENT_TYPES=['IN_MODIFY']
PATH=[/tmp] FILENAME=[test_file] EVENT_TYPES=['IN_OPEN']
PATH=[/tmp] FILENAME=[test_file] EVENT_TYPES=['IN_CLOSE_WRITE']
^CTraceback (most recent call last):
  File "inotify_test.py", line 18, in <module>
    _main()
  File "inotify_test.py", line 11, in _main
    for event in i.event_gen(yield_nones=False):
  File "/home/dustin/development/python/pyinotify/inotify/adapters.py", line 202, in event_gen
    events = self.__epoll.poll(block_duration_s)
KeyboardInterrupt

Note that this works quite nicely, but, in the event that you don't want to be driven by the loop, you can also provide a timeout and then even flatten the output of the generator directly to a list:

import inotify.adapters

def _main():
    i = inotify.adapters.Inotify()

    i.add_watch('/tmp')

    with open('/tmp/test_file', 'w'):
        pass

    events = i.event_gen(yield_nones=False, timeout_s=1)
    events = list(events)

    print(events)

if __name__ == '__main__':
    _main()

This will return everything that's happened since the last time you ran it (artificially formatted here):

[
    (_INOTIFY_EVENT(wd=1, mask=2, cookie=0, len=16), ['IN_MODIFY'], '/tmp', u'test_file'),
    (_INOTIFY_EVENT(wd=1, mask=32, cookie=0, len=16), ['IN_OPEN'], '/tmp', u'test_file'),
    (_INOTIFY_EVENT(wd=1, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], '/tmp', u'test_file')
]

Note that the event-loop will automatically register new directories to be watched, so, if you will create new directories and then potentially delete them, between calls, and are only retrieving the events in batches (like above) then you might experience issues. See the parameters for `event_gen()` for options to handle this scenario.

Recursive Watching

There is also the ability to add a recursive watch on a path.

Example:

i = inotify.adapters.InotifyTree('/tmp/watch_tree')

for event in i.event_gen():
    # Do stuff...

    pass

This will immediately recurse through the directory tree and add watches on all subdirectories. New directories will automatically have watches added for them and deleted directories will be cleaned-up.

The other differences from the standard functionality:

  • You can't remove a watch since watches are automatically managed.
  • Even if you provide a very restrictive mask that doesn't allow for directory create/delete events, the IN_ISDIR, IN_CREATE, and IN_DELETE flags will still be seen.

Notes

  • IMPORTANT: Recursively monitoring paths is not a functionality provided by the kernel. Rather, we artificially implement it. As directory-created events are received, we create watches for the child directories on-the-fly. This means that there is potential for a race condition: if a directory is created and a file or directory is created inside before you (using the event_gen() loop) have a chance to observe it, then you are going to have a problem: If it is a file, then you will miss the events related to its creation, but, if it is a directory, then not only will you miss those creation events but this library will also miss them and not be able to add a watch for them. If you are dealing with a large number of hierarchical directory creations and have the ability to be aware new directories via a secondary channel with some lead time before any files are populated into them, you can take advantage of this and call add_watch() manually. In this case there is limited value in using InotifyTree()/InotifyTree() instead of just Inotify() but this choice is left to you.
  • epoll is used to audit for inotify kernel events.
  • The earlier versions of this project had only partial Python 3 compatibility (string related). This required doing the string<->bytes conversions outside of this project. As of the current version, this has been fixed. However, this means that Python 3 users may experience breakages until this is compensated-for on their end. It will obviously be trivial for this project to detect the type of the arguments that are passed but there'd be no concrete way of knowing which type to return. Better to just fix it completely now and move forward.
  • You may also choose to pass the list of directories to watch via the paths parameter of the constructor. This would work best in situations where your list of paths is static.
  • Calling remove_watch() is not strictly necessary. The inotify resources is automatically cleaned-up, which would clean-up all watch resources as well.

Testing

It is possible to run tests using the setuptools test target:

$ python setup.py test

Or you can install nose and use that directly:

$ pip install nose

Then, call "test.sh" to run the tests:

$ ./test.sh
test__cycle (tests.test_inotify.TestInotify) ... ok
test__get_event_names (tests.test_inotify.TestInotify) ... ok
test__international_naming_python2 (tests.test_inotify.TestInotify) ... SKIP: Not in Python 2
test__international_naming_python3 (tests.test_inotify.TestInotify) ... ok
test__automatic_new_watches_on_existing_paths (tests.test_inotify.TestInotifyTree) ... ok
test__automatic_new_watches_on_new_paths (tests.test_inotify.TestInotifyTree) ... ok
test__cycle (tests.test_inotify.TestInotifyTree) ... ok
test__renames (tests.test_inotify.TestInotifyTree) ... ok
test__cycle (tests.test_inotify.TestInotifyTrees) ... ok

----------------------------------------------------------------------
Ran 9 tests in 12.039s

OK (SKIP=1)

pyinotify's People

Contributors

backbord avatar baszoetekouw avatar cbalfour avatar davidparsson avatar dsoprea avatar dwvisser avatar jessesuen avatar ml-bnr avatar xaf avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pyinotify's Issues

Inotify.__handle_inotify_event raises TypeError when attempting to handle an event in python3.

Stack trace:

Traceback (most recent call last):
File "/inotify/dev/test.py", line 50, in
_main()
File "/inotify/dev/test.py", line 38, in _main
for event in i.event_gen():
File "/inotify/inotify/adapters.py", line 162, in event_gen
in self.__handle_inotify_event(fd, event_type):
File "/inotify/inotify/adapters.py", line 114, in __handle_inotify_event
self.__buffer += b
TypeError: Can't convert 'bytes' object to str implicitly

The problem is that self.__buffer is initialized to '', which in python 3 is a unicode string.

Crashed in calls.py(32): _check_nonnegative

I made a simple program to monitor my home path:

import inotify.adapters

def _main():
    i = inotify.adapters.InotifyTree('/opt/test_user/')
    try:
        for event in i.event_gen():
            if event is None:
                continue
            (header, types, path, filename) = event
            if 'IN_ISDIR' not in types:
               print("PATH=[{}] FILENAME=[{}] EVENT_TYPES={}".format(
                 path, filename, types))
if __name__ == '__main__':
    _main()

I am getting the following error:

Traceback (most recent call last):
  File "sample2.py", line 26, in <module>
    _main()
  File "sample2.py", line 4, in _main
    i = inotify.adapters.InotifyTree('/opt/test_user/')
  File "/usr/local/lib/python2.7/dist-packages/inotify/adapters.py", line 340, in __init__
    self.__load_tree(path)
  File "/usr/local/lib/python2.7/dist-packages/inotify/adapters.py", line 354, in __load_tree
    for filename in os.listdir(current_path):

I had to run the program with SUDO though. But whenver I remove the following exception, all works fine:

def _check_nonnegative(result):
    if result == -1:
        raise InotifyError("Call failed (should not be -1): (%d)" % 
                           (result,))

Why was this check required? Is it good idea to remove it?

Regards,

Renato.

symbol not found on Python 3.7.3 OSX 10.14.2

import inotify.adapters


def _main():
    i = inotify.adapters.Inotify()

    i.add_watch('/tmp')

    with open('/tmp/test_file', 'w'):
        pass

    events = i.event_gen(yield_nones=False, timeout_s=1)
    events = list(events)

    print(events)


if __name__ == '__main__':
    _main()
$ python file-watcher.py
Traceback (most recent call last):
  File "file-watcher.py", line 1, in <module>
    import inotify.adapters
  File "/Users/sarit/.pyenv/versions/life/lib/python3.7/site-packages/inotify/adapters.py", line 11, in <module>
    import inotify.calls
  File "/Users/sarit/.pyenv/versions/life/lib/python3.7/site-packages/inotify/calls.py", line 39, in <module>
    inotify_init = _LIB.inotify_init
  File "/Users/sarit/.pyenv/versions/3.7.3/lib/python3.7/ctypes/__init__.py", line 369, in __getattr__
    func = self.__getitem__(name)
  File "/Users/sarit/.pyenv/versions/3.7.3/lib/python3.7/ctypes/__init__.py", line 374, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x11816fe90, inotify_init): symbol not found

OSX: 1014.2
Python: 3.7.3
inotify==0.2.10

Any idea?

Examples fail with errors

(venv) user@host:~/venv$ cat test_inotify.py 
#!/usr/bin/env python2
# -*- coding: utf-8 -*-
import inotify.adapters


def _main():
    i = inotify.adapters.Inotify()

    i.add_watch('/tmp')

    with open('/tmp/test_file', 'w'):
        pass

    events = i.event_gen(yield_nones=False, timeout_s=1)
    events = list(events)

    print(events)


if __name__ == '__main__':
    _main()

(venv) user@host:~/prod-export/pybuild23$ ./test_inotify.py 
Traceback (most recent call last):
  File "./test_inotify.py", line 21, in <module>
    _main()
  File "./test_inotify.py", line 14, in _main
    events = i.event_gen(yield_nones=False, timeout_s=1)
TypeError: event_gen() got an unexpected keyword argument 'yield_nones'
(venv) user@host:~/venv$ 

Removing the yield_nones=False results in another error.

Is this expected? Just a matter of documentation being out of date? I can read the source but I think most users will want to have very basic examples in the README working.

If I look through the source and make a simple example I will PR it, unless I am (somehow) doing this wrong...

Nose is a requirement?

I wrote some software that uses this module, and I now find myself needing to install that on a machine that has no direct connection to the internet.

So I did what I had been doing so far, namely use pip on my dev machine to download all modules and requirements, then upload that to the other machine and install with pip from the downloaded wheels.

To my surprise, though, when I downloaded this module, it also downloaded the nose module.
A quick check showed that yes indeed, nose is listed as a requirement in inotify/resources/requirements.txt, yet as I suspected, it isn't used anywhere.

Please remove this requirement as there is no need to have nose in a production environment...

Possible race condition when using InotifyTrees

Hello,
Thank you for this nice library it's really handy.
I think there could be race condition when using InotifyTrees().
The function build a list to use add_watch() on however as add_watch() do not check if the file or folder exists before adding the watch, it could crash in the process.
I happen to have a huge folder that is being watch but this race condition could happen.
I think there could be 2 possible solutions:

  • maybe add a check in add_watch() if file/folder exists
  • add a boolean to ignore missing folder and continue the instantiation when using InotifyTrees().
    What do you think about this ?

Inotify. add_watch raises an exception when passed a str path in python3.

Here is a stack trace:

Traceback (most recent call last):
File "/inotify/dev/test.py", line 50, in
_main()
File "/inotify/dev/test.py", line 35, in _main
i.add_watch('/tmp')
File "/inotify/inotify/adapters.py", line 65, in add_watch
wd = inotify.calls.inotify_add_watch(self.__inotify_fd, path, mask)
ctypes.ArgumentError: argument 2: <class 'TypeError'>: wrong type

inotify.calls.inotify_add_watch expects a byte-string for the path argument.

Seemingly getting code from 0.2.8 when 0.2.9 is the installed version

I have the following:

$ pip freeze
inotify==0.2.9
$ python
Python 2.7.14 (default, Jan 18 2018, 19:27:07) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import inotify.adapters
>>> import inspect
>>> print inspect.getsource(inotify.adapters.Inotify.event_gen)
    def event_gen(self):
        while True:
            block_duration_s = self.__get_block_duration()

            # Poll, but manage signal-related errors.

            try:
                events = self.__epoll.poll(block_duration_s)
            except IOError as e:
                if e.errno != EINTR:
                    raise

                continue

            # Process events.

            for fd, event_type in events:
                # (fd) looks to always match the inotify FD.

                for (header, type_names, path, filename) \
                        in self.__handle_inotify_event(fd, event_type):
                    yield (header, type_names, path, filename)

            yield None

>>> 

I originally noticed I was getting "unknown keyword argument" when trying to pass yield_nones=False to event_gen, which made me try to look up the source like this. It looks like the code that's being run is from version 0.2.8 on github: https://github.com/dsoprea/PyInotify/blob/0.2.8/inotify/adapters.py#L160

InotifyTree doesn't handle mkdir()+rmdir()

While using InotifyTree, I noticed that if you create and remove a directory very quickly, event_gen() fails. The reason for this is obvious: it sees the CREATE event and tries to install a new probe, which fails, as the target no longer exists.

InotifyTree should simply ignore such errors.

Create Dockerfile to support unit-test debugging/development from local and fix current CI issues

Currently, our version of Ubuntu is different from the CI version of Ubuntu and can explain at least one discrepancy in testing results. This is causing the CI to currently fail in testing. Specifically, we're getting duplicate IN_OPEN events on local and a IN_CLOSE_NOWRITE event in CI but not from local.

test__watch_list_of_paths (tests.test_inotify.TestInotify) ... 

ACTUAL:
(_INOTIFY_EVENT(wd=1, mask=256, cookie=0, len=16), ['IN_CREATE'], '/tmp/tmp7976pa7k/aa', 'seen_new_file')
(_INOTIFY_EVENT(wd=1, mask=32, cookie=0, len=16), ['IN_OPEN'], '/tmp/tmp7976pa7k/aa', 'seen_new_file')
(_INOTIFY_EVENT(wd=1, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], '/tmp/tmp7976pa7k/aa', 'seen_new_file')
(_INOTIFY_EVENT(wd=2, mask=256, cookie=0, len=16), ['IN_CREATE'], '/tmp/tmp7976pa7k/bb', 'seen_new_file2')
(_INOTIFY_EVENT(wd=2, mask=32, cookie=0, len=16), ['IN_OPEN'], '/tmp/tmp7976pa7k/bb', 'seen_new_file2')
(_INOTIFY_EVENT(wd=2, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], '/tmp/tmp7976pa7k/bb', 'seen_new_file2')
(_INOTIFY_EVENT(wd=1, mask=512, cookie=0, len=16), ['IN_DELETE'], '/tmp/tmp7976pa7k/aa', 'seen_new_file')

EXPECTED:
(_INOTIFY_EVENT(wd=1, mask=256, cookie=0, len=16), ['IN_CREATE'], '/tmp/tmp7976pa7k/aa', 'seen_new_file')
(_INOTIFY_EVENT(wd=1, mask=32, cookie=0, len=16), ['IN_OPEN'], '/tmp/tmp7976pa7k/aa', 'seen_new_file')
(_INOTIFY_EVENT(wd=1, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], '/tmp/tmp7976pa7k/aa', 'seen_new_file')
(_INOTIFY_EVENT(wd=2, mask=256, cookie=0, len=16), ['IN_CREATE'], '/tmp/tmp7976pa7k/bb', 'seen_new_file2')
(_INOTIFY_EVENT(wd=2, mask=32, cookie=0, len=16), ['IN_OPEN'], '/tmp/tmp7976pa7k/bb', 'seen_new_file2')
(_INOTIFY_EVENT(wd=2, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], '/tmp/tmp7976pa7k/bb', 'seen_new_file2')
(_INOTIFY_EVENT(wd=1, mask=512, cookie=0, len=16), ['IN_DELETE'], '/tmp/tmp7976pa7k/aa', 'seen_new_file')
(_INOTIFY_EVENT(wd=2, mask=32, cookie=0, len=16), ['IN_OPEN'], '/tmp/tmp7976pa7k/bb', 'seen_new_file2')
(_INOTIFY_EVENT(wd=2, mask=16, cookie=0, len=16), ['IN_CLOSE_NOWRITE'], '/tmp/tmp7976pa7k/bb', 'seen_new_file2')

I initially wanted to add some jitter-correction to drop similar events within a near timeframe to each other, but I don't think this can be done safely.

pytest Failed under CentOS7/Python3.8

[root@localhost tests]# /opt/python38/bin/pytest
======================================================================== test session starts ========================================================================
platform linux -- Python 3.8.3, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /root/libs/PyInotify-0.2.10
plugins: cov-2.8.1, forked-1.1.3, xdist-1.32.0
collected 9 items

test_inotify.py ..s..FFF.                                                                                                                                     [100%]

============================================================================= FAILURES ==============================================================================
_____________________________________________________ TestInotifyTree.test__automatic_new_watches_on_new_paths ______________________________________________________

self = <tests.test_inotify.TestInotifyTree testMethod=test__automatic_new_watches_on_new_paths>

    def test__automatic_new_watches_on_new_paths(self):

        # Tests that watches are actively established as new folders are
        # created.

        with inotify.test_support.temp_path() as path:
            i = inotify.adapters.InotifyTree(path)

            path1 = os.path.join(path, 'folder1')
            path2 = os.path.join(path1, 'folder2')

            os.mkdir(path1)

            events = self.__read_all_events(i)

            expected = [
                (inotify.adapters._INOTIFY_EVENT(wd=1, mask=1073742080, cookie=0, len=16), ['IN_ISDIR', 'IN_CREATE'], path, 'folder1'),
            ]

>           self.assertEquals(events, expected)
E           AssertionError: Lists differ: [(_IN[49 chars]16), ['IN_CREATE', 'IN_ISDIR'], '/tmp/tmpn2c40svx', 'folder1')] != [(_IN[49 chars]16), ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmpn2c40svx', 'folder1')]
E
E           First differing element 0:
E           (_INO[47 chars]=16), ['IN_CREATE', 'IN_ISDIR'], '/tmp/tmpn2c40svx', 'folder1')
E           (_INO[47 chars]=16), ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmpn2c40svx', 'folder1')
E
E             [(_INOTIFY_EVENT(wd=1, mask=1073742080, cookie=0, len=16),
E           -   ['IN_CREATE', 'IN_ISDIR'],
E           +   ['IN_ISDIR', 'IN_CREATE'],
E               '/tmp/tmpn2c40svx',
E               'folder1')]

test_inotify.py:298: AssertionError
____________________________________________________________________ TestInotifyTree.test__cycle ____________________________________________________________________

self = <tests.test_inotify.TestInotifyTree testMethod=test__cycle>

    def test__cycle(self):
        with inotify.test_support.temp_path() as path:
            path1 = os.path.join(path, 'aa')
            os.mkdir(path1)

            path2 = os.path.join(path, 'bb')
            os.mkdir(path2)

            i = inotify.adapters.InotifyTree(path)

            with open('seen_new_file1', 'w'):
                pass

            with open(os.path.join(path1, 'seen_new_file2'), 'w'):
                pass

            with open(os.path.join(path2, 'seen_new_file3'), 'w'):
                pass

            os.remove(os.path.join(path, 'seen_new_file1'))
            os.remove(os.path.join(path1, 'seen_new_file2'))
            os.remove(os.path.join(path2, 'seen_new_file3'))

            os.rmdir(path1)
            os.rmdir(path2)

            events = self.__read_all_events(i)

            expected = [
                (inotify.adapters._INOTIFY_EVENT(wd=1, mask=256, cookie=0, len=16), ['IN_CREATE'], path, 'seen_new_file1'),
                (inotify.adapters._INOTIFY_EVENT(wd=1, mask=32, cookie=0, len=16), ['IN_OPEN'], path, 'seen_new_file1'),
                (inotify.adapters._INOTIFY_EVENT(wd=1, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], path, 'seen_new_file1'),

                (inotify.adapters._INOTIFY_EVENT(wd=2, mask=256, cookie=0, len=16), ['IN_CREATE'], path1, 'seen_new_file2'),
                (inotify.adapters._INOTIFY_EVENT(wd=2, mask=32, cookie=0, len=16), ['IN_OPEN'], path1, 'seen_new_file2'),
                (inotify.adapters._INOTIFY_EVENT(wd=2, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], path1, 'seen_new_file2'),

                (inotify.adapters._INOTIFY_EVENT(wd=3, mask=256, cookie=0, len=16), ['IN_CREATE'], path2, 'seen_new_file3'),
                (inotify.adapters._INOTIFY_EVENT(wd=3, mask=32, cookie=0, len=16), ['IN_OPEN'], path2, 'seen_new_file3'),
                (inotify.adapters._INOTIFY_EVENT(wd=3, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], path2, 'seen_new_file3'),

                (inotify.adapters._INOTIFY_EVENT(wd=1, mask=512, cookie=0, len=16), ['IN_DELETE'], path, 'seen_new_file1'),
                (inotify.adapters._INOTIFY_EVENT(wd=2, mask=512, cookie=0, len=16), ['IN_DELETE'], path1, 'seen_new_file2'),
                (inotify.adapters._INOTIFY_EVENT(wd=3, mask=512, cookie=0, len=16), ['IN_DELETE'], path2, 'seen_new_file3'),

                (inotify.adapters._INOTIFY_EVENT(wd=2, mask=1024, cookie=0, len=0), ['IN_DELETE_SELF'], path1, ''),
                (inotify.adapters._INOTIFY_EVENT(wd=2, mask=32768, cookie=0, len=0), ['IN_IGNORED'], path1, ''),
                (inotify.adapters._INOTIFY_EVENT(wd=1, mask=1073742336, cookie=0, len=16), ['IN_ISDIR', 'IN_DELETE'], path, 'aa'),

                (inotify.adapters._INOTIFY_EVENT(wd=3, mask=1024, cookie=0, len=0), ['IN_DELETE_SELF'], path2, ''),
                (inotify.adapters._INOTIFY_EVENT(wd=3, mask=32768, cookie=0, len=0), ['IN_IGNORED'], path2, ''),
                (inotify.adapters._INOTIFY_EVENT(wd=1, mask=1073742336, cookie=0, len=16), ['IN_ISDIR', 'IN_DELETE'], path, 'bb'),
            ]

>           self.assertEquals(events, expected)
E           AssertionError: Lists differ: [(_IN[1294 chars]T(wd=1, mask=1073742336, cookie=0, len=16), ['[548 chars] '')] != [(_IN[1294 chars]T(wd=2, mask=1024, cookie=0, len=0), ['IN_DELE[548 chars]bb')]
E
E           First differing element 12:
E           (_INOTIFY_EVENT(wd=1, mask=1073742336, cookie=0, len=16), ['[45 chars]'aa')
E           (_INOTIFY_EVENT(wd=2, mask=1024, cookie=0, len=0), ['IN_DELE[32 chars], '')
E
E             [(_INOTIFY_EVENT(wd=1, mask=256, cookie=0, len=16),
E               ['IN_CREATE'],
E               '/tmp/tmpkw8e5ko_',
E               'seen_new_file1'),
E              (_INOTIFY_EVENT(wd=1, mask=32, cookie=0, len=16),
E               ['IN_OPEN'],
E               '/tmp/tmpkw8e5ko_',
E               'seen_new_file1'),
E              (_INOTIFY_EVENT(wd=1, mask=8, cookie=0, len=16),
E               ['IN_CLOSE_WRITE'],
E               '/tmp/tmpkw8e5ko_',
E               'seen_new_file1'),
E              (_INOTIFY_EVENT(wd=2, mask=256, cookie=0, len=16),
E               ['IN_CREATE'],
E               '/tmp/tmpkw8e5ko_/aa',
E               'seen_new_file2'),
E              (_INOTIFY_EVENT(wd=2, mask=32, cookie=0, len=16),
E               ['IN_OPEN'],
E               '/tmp/tmpkw8e5ko_/aa',
E               'seen_new_file2'),
E              (_INOTIFY_EVENT(wd=2, mask=8, cookie=0, len=16),
E               ['IN_CLOSE_WRITE'],
E               '/tmp/tmpkw8e5ko_/aa',
E               'seen_new_file2'),
E              (_INOTIFY_EVENT(wd=3, mask=256, cookie=0, len=16),
E               ['IN_CREATE'],
E               '/tmp/tmpkw8e5ko_/bb',
E               'seen_new_file3'),
E              (_INOTIFY_EVENT(wd=3, mask=32, cookie=0, len=16),
E               ['IN_OPEN'],
E               '/tmp/tmpkw8e5ko_/bb',
E               'seen_new_file3'),
E              (_INOTIFY_EVENT(wd=3, mask=8, cookie=0, len=16),
E               ['IN_CLOSE_WRITE'],
E               '/tmp/tmpkw8e5ko_/bb',
E               'seen_new_file3'),
E              (_INOTIFY_EVENT(wd=1, mask=512, cookie=0, len=16),
E               ['IN_DELETE'],
E               '/tmp/tmpkw8e5ko_',
E               'seen_new_file1'),
E              (_INOTIFY_EVENT(wd=2, mask=512, cookie=0, len=16),
E               ['IN_DELETE'],
E               '/tmp/tmpkw8e5ko_/aa',
E               'seen_new_file2'),
E              (_INOTIFY_EVENT(wd=3, mask=512, cookie=0, len=16),
E               ['IN_DELETE'],
E               '/tmp/tmpkw8e5ko_/bb',
E               'seen_new_file3'),
E           -  (_INOTIFY_EVENT(wd=1, mask=1073742336, cookie=0, len=16),
E           -   ['IN_DELETE', 'IN_ISDIR'],
E           -   '/tmp/tmpkw8e5ko_',
E           -   'aa'),
E           -  (_INOTIFY_EVENT(wd=1, mask=1073742336, cookie=0, len=16),
E           -   ['IN_DELETE', 'IN_ISDIR'],
E           -   '/tmp/tmpkw8e5ko_',
E           -   'bb'),
E              (_INOTIFY_EVENT(wd=2, mask=1024, cookie=0, len=0),
E               ['IN_DELETE_SELF'],
E               '/tmp/tmpkw8e5ko_/aa',
E               ''),
E              (_INOTIFY_EVENT(wd=2, mask=32768, cookie=0, len=0),
E               ['IN_IGNORED'],
E               '/tmp/tmpkw8e5ko_/aa',
E               ''),
E           +  (_INOTIFY_EVENT(wd=1, mask=1073742336, cookie=0, len=16),
E           +   ['IN_ISDIR', 'IN_DELETE'],
E           +   '/tmp/tmpkw8e5ko_',
E           +   'aa'),
E              (_INOTIFY_EVENT(wd=3, mask=1024, cookie=0, len=0),
E               ['IN_DELETE_SELF'],
E               '/tmp/tmpkw8e5ko_/bb',
E               ''),
E              (_INOTIFY_EVENT(wd=3, mask=32768, cookie=0, len=0),
E               ['IN_IGNORED'],
E               '/tmp/tmpkw8e5ko_/bb',
E           +   ''),
E           +  (_INOTIFY_EVENT(wd=1, mask=1073742336, cookie=0, len=16),
E           +   ['IN_ISDIR', 'IN_DELETE'],
E           +   '/tmp/tmpkw8e5ko_',
E           -   '')]
E           +   'bb')]
E           ?    ++

test_inotify.py:213: AssertionError
___________________________________________________________________ TestInotifyTree.test__renames ___________________________________________________________________

self = <tests.test_inotify.TestInotifyTree testMethod=test__renames>

    def test__renames(self):

        # Since we're not reading the events one at a time in a loop and
        # removing or renaming folders will flush any queued events, we have to
        # group things in order to check things first before such operations.

        with inotify.test_support.temp_path() as path:
            i = inotify.adapters.InotifyTree(path)

            old_path = os.path.join(path, 'old_folder')
            new_path = os.path.join(path, 'new_folder')

            os.mkdir(old_path)

            events1 = self.__read_all_events(i)

            expected = [
                (inotify.adapters._INOTIFY_EVENT(wd=1, mask=1073742080, cookie=events1[0][0].cookie, len=16), ['IN_ISDIR', 'IN_CREATE'], path, 'old_folder'),
            ]

>           self.assertEquals(events1, expected)
E           AssertionError: Lists differ: [(_IN[52 chars], ['IN_CREATE', 'IN_ISDIR'], '/tmp/tmpaawvmb4h', 'old_folder')] != [(_IN[52 chars], ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmpaawvmb4h', 'old_folder')]
E
E           First differing element 0:
E           (_INO[50 chars]), ['IN_CREATE', 'IN_ISDIR'], '/tmp/tmpaawvmb4h', 'old_folder')
E           (_INO[50 chars]), ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmpaawvmb4h', 'old_folder')
E
E             [(_INOTIFY_EVENT(wd=1, mask=1073742080, cookie=0, len=16),
E           -   ['IN_CREATE', 'IN_ISDIR'],
E           +   ['IN_ISDIR', 'IN_CREATE'],
E               '/tmp/tmpaawvmb4h',
E               'old_folder')]

test_inotify.py:235: AssertionError
========================================================================= warnings summary ==========================================================================
tests/test_inotify.py::TestInotify::test__cycle
  /root/libs/PyInotify-0.2.10/tests/test_inotify.py:123: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(events, expected)

tests/test_inotify.py::TestInotify::test__cycle
  /root/libs/PyInotify-0.2.10/tests/test_inotify.py:133: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(events, [])

tests/test_inotify.py::TestInotify::test__get_event_names
  /root/libs/PyInotify-0.2.10/tests/test_inotify.py:146: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(names, all_names)

tests/test_inotify.py::TestInotify::test__international_naming_python3
  /root/libs/PyInotify-0.2.10/tests/test_inotify.py:48: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(events, expected)

tests/test_inotify.py::TestInotifyTree::test__automatic_new_watches_on_existing_paths
  /root/libs/PyInotify-0.2.10/tests/test_inotify.py:350: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(events, expected)

tests/test_inotify.py::TestInotifyTree::test__automatic_new_watches_on_new_paths
  /root/libs/PyInotify-0.2.10/tests/test_inotify.py:298: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(events, expected)

tests/test_inotify.py::TestInotifyTree::test__cycle
  /root/libs/PyInotify-0.2.10/tests/test_inotify.py:213: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(events, expected)

tests/test_inotify.py::TestInotifyTree::test__renames
  /root/libs/PyInotify-0.2.10/tests/test_inotify.py:235: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(events1, expected)

tests/test_inotify.py::TestInotifyTrees::test__cycle
  /root/libs/PyInotify-0.2.10/tests/test_inotify.py:391: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(events, expected)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
====================================================================== short test summary info ======================================================================
FAILED test_inotify.py::TestInotifyTree::test__automatic_new_watches_on_new_paths - AssertionError: Lists differ: [(_IN[49 chars]16), ['IN_CREATE', 'IN_ISDIR'], '...
FAILED test_inotify.py::TestInotifyTree::test__cycle - AssertionError: Lists differ: [(_IN[1294 chars]T(wd=1, mask=1073742336, cookie=0, len=16), ['[548 chars] ''...
FAILED test_inotify.py::TestInotifyTree::test__renames - AssertionError: Lists differ: [(_IN[52 chars], ['IN_CREATE', 'IN_ISDIR'], '/tmp/tmpaawvmb4h', 'old_folder...
======================================================== 3 failed, 5 passed, 1 skipped, 9 warnings in 8.61s =========================================================
[root@localhost tests]#

Recursive watching fails on directory deletion

The following scenario does not work as expected:

  • Watching a directory recursively
  • Creating a directory inside the watched directory
  • create files inside the new directory
  • delete the created directory
  • everything until now works fine
  • create the directory again
    -> Path already being watched: [path of the directory]
  • everything inside this directory is not watched anymore

Recursive: mkdir -p a/b/c/d/e/f/g/ fails?

I guess the recursive monitoring of changes this mkdir -p a/b/c/d/e/f/g/ fails, since there is a race condition.

AFAIK this race condition exists always and there is not way to work around this issue in python.

If this is true, it would be nice, if you could note this in the README.

Crash when directory created over SSH.

I am watching a directory tree "sync", when a file changes in that directory I trigger a HTTP request to 2 other machines that then perform a unison/rsync from that directory. This all works fine unless a directory is created on the remote. When the sync happens Inotify crashes with
"2019-01-31 08:41:55,529 - root - Event Error 2019-01-31 08:41:55,529 - root - Call failed (should not be -1): (-1) ERRNO=(0)"

This only happens when new directory's are created by unison over SSH.

parallelization of events notification

Hi,

The Example provided works great to get notifications in a serial manner. However, it is not good for large images.
For example, I am uploading a set of pictures. Pictures that are 20MB each and it takes 2 seconds each time to finish the notifications for each files. Then it goes to next file and so on, till all the of the upload.
I tried to add parallelization of events notification without success :
here is my code :

`
import logging
import threading
import inotify.adapters

 _DEFAULT_LOG_FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
 
 _LOGGER = logging.getLogger(__name__)
 
 def _configure_logging():
     _LOGGER.setLevel(logging.DEBUG)
 
     ch = logging.StreamHandler()
 
     formatter = logging.Formatter(_DEFAULT_LOG_FORMAT)
     ch.setFormatter(formatter)
 
     _LOGGER.addHandler(ch)
 
 
 #def PopUpMessage (event):
 #    (header, type_names, watch_path, filename) = event
 #    _LOGGER.info("WD=(%d) MASK=(%d) COOKIE=(%d) LEN=(%d) MASK->NAMES=%s "
 #        "WATCH-PATH=[%s] FILENAME=[%s]",
 #        header.wd, header.mask, header.cookie, header.len, type_names,
 #        watch_path.decode('utf-8'), filename.decode('utf-8'))
 
 def PopUpMessage (event):
     if event is not None:
         (header, type_names, watch_path, filename) = event
         _LOGGER.info("WD=(%d) MASK=(%d) COOKIE=(%d) LEN=(%d) MASK->NAMES=%s "
             "WATCH-PATH=[%s] FILENAME=[%s]",
             header.wd, header.mask, header.cookie, header.len, type_names,
             watch_path.decode('utf-8'), filename.decode('utf-8'))
 
 
 def _main():
     i = inotify.adapters.Inotify()
 
     i.add_watch(b'/PARA')
 
     try:
         threads = []
         while True: 
             for event in i.event_gen():
 
                 #if event is not None:
                 #(header, type_names, watch_path, filename) = event
                 #thread = threading.Thread
                 ti= threading.Thread(target=PopUpMessage(event))
                 threads += [ti]
                 ti.start()
             for x in threads:
                 x.join()
 
     finally:
         i.remove_watch(b'/PARA')
 
 if __name__ == '__main__':
     _configure_logging()
     _main()`

inotify can not run properly under alpine linux

When I user inotify under alpine linux, it shows such error:
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.6/site-packages/inotify/adapters.py", line 10, in
import inotify.calls
File "/usr/lib/python3.6/site-packages/inotify/calls.py", line 55, in
errno = _LIB.errno
File "/usr/lib/python3.6/ctypes/init.py", line 361, in getattr
func = self.getitem(name)
File "/usr/lib/python3.6/ctypes/init.py", line 366, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: Symbol not found: errno

New release?

It's been a long time since last release. In the meantime, some bugs got fixed, so I'd be happy to see a new release. Could we get one, please? :)

Use of pathname instead of watch descriptor is problematic

Using the path as the handle to refer to watches causes issues in some situations, because the pathname the watch was created against is not guaranteed to be unique/distinct for the life of the watch.

For example, take the following scenario:

  1. Create a watch against /dir/foo
  2. Somebody does mv /dir/foo /dir/bar (at this point, the watch created in step 1 is still active, still valid, and still watching the file which used to be /dir/foo, even though it's name has now changed)
  3. Somebody creates a new /dir/foo file.
  4. The application wants to keep watching the old file, but also open a new watch on the new /dir/foo file.

In this situation, the inotify library gets hopelessly confused (and even if it didn't, it doesn't return adequate information from watch events for the app to be able to figure out which watch is indicating what).

This is actually causing problems for me as I'm trying to write an application to monitor log files (which regularly get rotated in this way).

My recommendation would be that Inotify.add_watch should create a unique "Watch object" for each active watch, which it returns to the caller, and would then be provided as part of watch events (and could be supplied as an argument to remove_watch, etc.) to be able to uniquely distinguish between watches, even if multiple ones have the same associated path.

Under python 3.4 constants.py gives an error at import

The way octal is used in python3.4 has changed. Now a prefix of 0o needs to be added.

Python 3.4.0 (default, Jun 19 2015, 14:20:21)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.

import inotify.adapters
Traceback (most recent call last):
File "", line 1, in
File "/home/dick/.virtualenvs/cctools/lib/python3.4/site-packages/inotify/adapters.py", line 7, in
import inotify.constants
File "/home/dick/.virtualenvs/cctools/lib/python3.4/site-packages/inotify/constants.py", line 3
IN_CLOEXEC = 02000000
^
SyntaxError: invalid token

remove_watch functionality

Few points regarding remove_watch functionality.

  1. remove_watch should propagate superficial argument to remove_watch_with_id (but I see #57 is already in place for that)
  • for this this the Tree-Handling would need probably a fix (use superficial only for currently unhandeled but probably wanted DELETE event #58 / #51 but not for MOVED_FROM as this does not unregister the watch from inotify see also #46 for a possible result...)
  1. I would like a public method to remove a wd only from underlying inotify (inotify.calls.inotify_rm_watch) to have possibility to remove the watch from one thread and watching thread is propagating the change which is does not using public remove_watch as it suppress the event if diretory name cannot be looked up) - possibly just a parameter (why not reuse superficial so True = only tracking information, False = tracking information + watch, None = watch only...)
  2. method remove_watch_with_id should not be public (without _) in current form as the self.__watches is not cleanup up using this method and there is no public method to only cleanup that - but best to make both existing methods removing both forward and reverse lookup entries for wd (and if superficial is False also remove the wd from underlying inotify)
  3. think about handling IN_IGNORE events received (in this case the the wd was unregistered automatically by underlying inotify) - so IN_IGNORE could cause automatic remove_watch with superficial set to True (sure that could be left up to the user. but at least there should be working methods for this case, currently is neighter - one does not support superficial the other does not remove forward entry from map..) - anyway if this would be implemented this could break the

Btw. in _BaseTree event_gen is another bug... The IN_MOVED_TO is handled twice so it will log a warning in that case...

Certain TestInotifyTree tests consistently failing

I see the following failures on my Ubuntu 18.04 VM, running in a virtualenv (either Py2 or Py3):

FAIL: test__cycle (tests.test_inotify.TestInotifyTree)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dale/Git/PyInotify/tests/test_inotify.py", line 213, in test__cycle
    self.assertEquals(events, expected)
AssertionError: Lists differ: [(_IN[325 chars]T(wd=3, mask=256, cookie=0, len=16), ['IN_CREA[1517 chars]bb')] != [(_IN[325 chars]T(wd=2, mask=256, cookie=0, len=16), ['IN_CREA[1517 chars]bb')]

First differing element 3:
(_INOTIFY_EVENT(wd=3, mask=256, cookie=0, len=16), ['IN_CREA[41 chars]le2')
(_INOTIFY_EVENT(wd=2, mask=256, cookie=0, len=16), ['IN_CREA[41 chars]le2')

  [(_INOTIFY_EVENT(wd=1, mask=256, cookie=0, len=16),
    ['IN_CREATE'],
    '/tmp/tmprcjv466m',
    'seen_new_file1'),
   (_INOTIFY_EVENT(wd=1, mask=32, cookie=0, len=16),
    ['IN_OPEN'],
    '/tmp/tmprcjv466m',
    'seen_new_file1'),
   (_INOTIFY_EVENT(wd=1, mask=8, cookie=0, len=16),
    ['IN_CLOSE_WRITE'],
    '/tmp/tmprcjv466m',
    'seen_new_file1'),
-  (_INOTIFY_EVENT(wd=3, mask=256, cookie=0, len=16),
?                     ^

+  (_INOTIFY_EVENT(wd=2, mask=256, cookie=0, len=16),
?                     ^

    ['IN_CREATE'],
    '/tmp/tmprcjv466m/aa',
    'seen_new_file2'),
-  (_INOTIFY_EVENT(wd=3, mask=32, cookie=0, len=16),
?                     ^

+  (_INOTIFY_EVENT(wd=2, mask=32, cookie=0, len=16),
?                     ^

    ['IN_OPEN'],
    '/tmp/tmprcjv466m/aa',
    'seen_new_file2'),
-  (_INOTIFY_EVENT(wd=3, mask=8, cookie=0, len=16),
?                     ^

+  (_INOTIFY_EVENT(wd=2, mask=8, cookie=0, len=16),
?                     ^

    ['IN_CLOSE_WRITE'],
    '/tmp/tmprcjv466m/aa',
    'seen_new_file2'),
-  (_INOTIFY_EVENT(wd=2, mask=256, cookie=0, len=16),
?                     ^

+  (_INOTIFY_EVENT(wd=3, mask=256, cookie=0, len=16),
?                     ^

    ['IN_CREATE'],
    '/tmp/tmprcjv466m/bb',
    'seen_new_file3'),
-  (_INOTIFY_EVENT(wd=2, mask=32, cookie=0, len=16),
?                     ^

+  (_INOTIFY_EVENT(wd=3, mask=32, cookie=0, len=16),
?                     ^

    ['IN_OPEN'],
    '/tmp/tmprcjv466m/bb',
    'seen_new_file3'),
-  (_INOTIFY_EVENT(wd=2, mask=8, cookie=0, len=16),
?                     ^

+  (_INOTIFY_EVENT(wd=3, mask=8, cookie=0, len=16),
?                     ^

    ['IN_CLOSE_WRITE'],
    '/tmp/tmprcjv466m/bb',
    'seen_new_file3'),
   (_INOTIFY_EVENT(wd=1, mask=512, cookie=0, len=16),
    ['IN_DELETE'],
    '/tmp/tmprcjv466m',
    'seen_new_file1'),
-  (_INOTIFY_EVENT(wd=3, mask=512, cookie=0, len=16),
?                     ^

+  (_INOTIFY_EVENT(wd=2, mask=512, cookie=0, len=16),
?                     ^

    ['IN_DELETE'],
    '/tmp/tmprcjv466m/aa',
    'seen_new_file2'),
-  (_INOTIFY_EVENT(wd=2, mask=512, cookie=0, len=16),
?                     ^

+  (_INOTIFY_EVENT(wd=3, mask=512, cookie=0, len=16),
?                     ^

    ['IN_DELETE'],
    '/tmp/tmprcjv466m/bb',
    'seen_new_file3'),
-  (_INOTIFY_EVENT(wd=3, mask=1024, cookie=0, len=0),
?                     ^

+  (_INOTIFY_EVENT(wd=2, mask=1024, cookie=0, len=0),
?                     ^

    ['IN_DELETE_SELF'],
    '/tmp/tmprcjv466m/aa',
    ''),
-  (_INOTIFY_EVENT(wd=3, mask=32768, cookie=0, len=0),
?                     ^

+  (_INOTIFY_EVENT(wd=2, mask=32768, cookie=0, len=0),
?                     ^

    ['IN_IGNORED'],
    '/tmp/tmprcjv466m/aa',
    ''),
   (_INOTIFY_EVENT(wd=1, mask=1073742336, cookie=0, len=16),
-   ['IN_DELETE', 'IN_ISDIR'],
+   ['IN_ISDIR', 'IN_DELETE'],
    '/tmp/tmprcjv466m',
    'aa'),
-  (_INOTIFY_EVENT(wd=2, mask=1024, cookie=0, len=0),
?                     ^

+  (_INOTIFY_EVENT(wd=3, mask=1024, cookie=0, len=0),
?                     ^

    ['IN_DELETE_SELF'],
    '/tmp/tmprcjv466m/bb',
    ''),
-  (_INOTIFY_EVENT(wd=2, mask=32768, cookie=0, len=0),
?                     ^

+  (_INOTIFY_EVENT(wd=3, mask=32768, cookie=0, len=0),
?                     ^

    ['IN_IGNORED'],
    '/tmp/tmprcjv466m/bb',
    ''),
   (_INOTIFY_EVENT(wd=1, mask=1073742336, cookie=0, len=16),
-   ['IN_DELETE', 'IN_ISDIR'],
+   ['IN_ISDIR', 'IN_DELETE'],
    '/tmp/tmprcjv466m',
    'bb')]
-------------------- >> begin captured logging << --------------------
inotify.adapters: DEBUG: Inotify handle is (3).
inotify.adapters: DEBUG: Adding initial watches on tree: [/tmp/tmprcjv466m]
inotify.adapters: DEBUG: Adding watch: [/tmp/tmprcjv466m]
inotify.adapters: DEBUG: Added watch (1): [/tmp/tmprcjv466m]
inotify.adapters: DEBUG: Adding watch: [/tmp/tmprcjv466m/bb]
inotify.adapters: DEBUG: Added watch (2): [/tmp/tmprcjv466m/bb]
inotify.adapters: DEBUG: Adding watch: [/tmp/tmprcjv466m/aa]
inotify.adapters: DEBUG: Added watch (3): [/tmp/tmprcjv466m/aa]
inotify.adapters: DEBUG: Events received from epoll: ['IN_ACCESS']
inotify.adapters: DEBUG: Events received in stream: ['IN_CREATE']
inotify.adapters: DEBUG: Events received in stream: ['IN_OPEN']
inotify.adapters: DEBUG: Events received in stream: ['IN_CLOSE_WRITE']
inotify.adapters: DEBUG: Events received in stream: ['IN_CREATE']
inotify.adapters: DEBUG: Events received in stream: ['IN_OPEN']
inotify.adapters: DEBUG: Events received in stream: ['IN_CLOSE_WRITE']
inotify.adapters: DEBUG: Events received in stream: ['IN_CREATE']
inotify.adapters: DEBUG: Events received in stream: ['IN_OPEN']
inotify.adapters: DEBUG: Events received in stream: ['IN_CLOSE_WRITE']
inotify.adapters: DEBUG: Events received in stream: ['IN_DELETE']
inotify.adapters: DEBUG: Events received in stream: ['IN_DELETE']
inotify.adapters: DEBUG: Events received in stream: ['IN_DELETE']
inotify.adapters: DEBUG: Events received in stream: ['IN_DELETE_SELF']
inotify.adapters: DEBUG: Events received in stream: ['IN_IGNORED']
inotify.adapters: DEBUG: Events received in stream: ['IN_DELETE', 'IN_ISDIR']
inotify.adapters: DEBUG: Events received in stream: ['IN_DELETE_SELF']
inotify.adapters: DEBUG: Events received in stream: ['IN_IGNORED']
inotify.adapters: DEBUG: Events received in stream: ['IN_DELETE', 'IN_ISDIR']
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test__renames (tests.test_inotify.TestInotifyTree)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dale/Git/PyInotify/tests/test_inotify.py", line 235, in test__renames
    self.assertEquals(events1, expected)
AssertionError: Lists differ: [(_IN[52 chars], ['IN_CREATE', 'IN_ISDIR'], '/tmp/tmp6orzx4kp', 'old_folder')] != [(_IN[52 chars], ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmp6orzx4kp', 'old_folder')]

First differing element 0:
(_INO[50 chars]), ['IN_CREATE', 'IN_ISDIR'], '/tmp/tmp6orzx4kp', 'old_folder')
(_INO[50 chars]), ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmp6orzx4kp', 'old_folder')

  [(_INOTIFY_EVENT(wd=1, mask=1073742080, cookie=0, len=16),
-   ['IN_CREATE', 'IN_ISDIR'],
+   ['IN_ISDIR', 'IN_CREATE'],
    '/tmp/tmp6orzx4kp',
    'old_folder')]
-------------------- >> begin captured logging << --------------------

errno is always 0

Hello there,

While using PyInotify I've noticed that the ERRNO value returned by InotifyError is always equal to zero:

In [1]: import os

In [2]: path = "/foo"

In [3]: os.path.exists(path)
Out[3]: False

In [4]: import inotify.adapters

In [5]: i = inotify.adapters.Inotify()

In [6]: i.add_watch(path)
---------------------------------------------------------------------------
InotifyError                              Traceback (most recent call last)
<ipython-input-6-5bee764f928e> in <module>()
----> 1 i.add_watch(path)

/home/florent/.local/lib/python2.7/site-packages/inotify/adapters.pyc in add_watch(self, path, mask)
     65         _LOGGER.debug("Adding watch: [%s]", path)
     66
---> 67         wd = inotify.calls.inotify_add_watch(self.__inotify_fd, path, mask)
     68         _LOGGER.debug("Added watch (%d): [%s]", wd, path)
     69

/home/florent/.local/lib/python2.7/site-packages/inotify/calls.pyc in _check_nonnegative(result)
     33     if result == -1:
     34         raise InotifyError("Call failed (should not be -1): (%d)" %
---> 35                            (result,))
     36
     37     return result

InotifyError: Call failed (should not be -1): (-1) ERRNO=(0)    

In this specific case I would expect ERRNO to be equal to ENOENT.

I am using python 2.7 and Jessie:

[21:11:00 :~]$ python --version
Python 2.7.9
[21:11:02 :~]$  cat /etc/debian_version
8.2

event_names should be a SET. Cleaner and testsuite more robust

Currently many tests are failing because event_names like "['IN_CREATE', 'IN_ISDIR']" do not garantee ordering, so the "assertEqual()" fails.

This could be solved using a SET, and the result is cleaner because order names in an event is irrelevant (they are actually flags).

When running the tests now I get trivial errors like:

...
======================================================================
FAIL: test__renames (tests.test_inotify.TestInotifyTree)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/PyInotify/tests/test_inotify.py", line 235, in test__renames
    self.assertEquals(events1, expected)
AssertionError: Lists differ: [(_IN[52 chars], ['IN_CREATE', 'IN_ISDIR'], '/tmp/tmpfgwdg1uf', 'old_folder')] != [(_IN[52 chars], ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmpfgwdg1uf', 'old_folder')]

First differing element 0:
(_INO[50 chars]), ['IN_CREATE', 'IN_ISDIR'], '/tmp/tmpfgwdg1uf', 'old_folder')
(_INO[50 chars]), ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmpfgwdg1uf', 'old_folder')

  [(_INOTIFY_EVENT(wd=1, mask=1073742080, cookie=0, len=16),
-   ['IN_CREATE', 'IN_ISDIR'],
+   ['IN_ISDIR', 'IN_CREATE'],
    '/tmp/tmpfgwdg1uf'
...

I could submit a patch if you want.

[opt] use scandir to shorten initialization time of Inotify

I want to use inotify to monitor video files on cdn server, and it takes too long time to initialize the InotifyTrees when i run the demo script (about 1061070 files)
I notice that os.listdir is used in the code, is there any possibility that we can use scandir.listdir (it is said that scandir will be merged to Python3 official in next release) to optimize the initialization speed?

InotifyTrees raises exception in log call for bytes argument (expecting str)

Reproduce:

from inotify.adapters import InotifyTrees

inots = InotifyTrees([
    b"something/",
    b"else/"
])

Results in:

Traceback (most recent call last):
  File "demo.py", line 4, in <module>
    b"output_files"
  File "/home/myles/software/conda/envs/dev35/lib/python3.5/site-packages/inotify/adapters.py", line 261, in __init__
    self.__load_trees(paths)
  File "/home/myles/software/conda/envs/dev35/lib/python3.5/site-packages/inotify/adapters.py", line 264, in __load_trees
    _LOGGER.debug("Adding initial watches on trees: [%s]", ",".join(paths))
TypeError: sequence item 0: expected str instance, bytes found

Using an Inotify object in select

I have reason to wait for either inotify events or stdin events and this code works:

    observer = inotify.adapters.Inotify()
    observer.add_watch(b'tmp')
    fd = observer._Inotify__inotify_fd
    [r, w, x] = select.select([sys.stdin, fd], [], [], 10)

But I would prefer this solution:

    observer = inotify.adapters.Inotify()
    observer.add_watch(b'tmp')
    [r, w, x] = select.select([sys.stdin, observer], [], [], 10)

I suspect it is a matter of implementing fileno() in Inotify:

   def fileno(self):
      return self.__inotify_fd

Add watch directly on pathlib instances

The add_watch function only supports strings when supplying the path to watch. Thus, if one uses Python 3.4's pathlib module for paths, it has to be converted to a string first when adding a watch:

path = pathlib.Path('a/path')
i = inotify.adapters.Inotify()
i.add_watch(str(path))

While minor, it could be useful to allow passing pathlib instances directly:

path = pathlib.Path('a/path')
i = inotify.adapters.Inotify()
i.add_watch(path)

question: how to watch for a specific event (IN_CLOSE_WRITE)

hi,

and thanks a lot for this library, so useful :) may I ask you the following;
how the original snippet that watches a directory (non-recursively) can be changed in order to
watch for a specific event only, for example,

for event in i.event_gen(yield_nones=False):
        (_, type_names, path, filename) = event

I do not wish to loop through type_names but I would like to get an event if and only if there is a IN_CLOSE_WRITE file event

if I am interested only in IN_CLOSE_WRITE file events, how can I I specify that ?
so that I can take action after that event takes place ?

Moving a previously removed folder crashing

Error

Traceback (most recent call last):
  File "../test.py", line 20, in <module>
    _main()
  File "../test.py", line 11, in _main
    for event in i.event_gen(yield_nones=False):
  File "/home/jvlarble/.local/lib/python3.6/site-packages/inotify/adapters.py", line 307, in event_gen
    self._i.remove_watch(full_path, superficial=True)
  File "/home/jvlarble/.local/lib/python3.6/site-packages/inotify/adapters.py", line 118, in remove_watch
    self.remove_watch_with_id(wd)
  File "/home/jvlarble/.local/lib/python3.6/site-packages/inotify/adapters.py", line 126, in remove_watch_with_id
    inotify.calls.inotify_rm_watch(self.__inotify_fd, wd)
  File "/home/jvlarble/.local/lib/python3.6/site-packages/inotify/calls.py", line 35, in _check_nonnegative
    (result,))
inotify.calls.InotifyError: Call failed (should not be -1): (-1) ERRNO=(0)

Source

import inotify.adapters

def _main():
    i = inotify.adapters.InotifyTree('.')

    #i.add_watch('.')

    #with open('/tmp/test_file', 'w'):
    #    pass

    for event in i.event_gen(yield_nones=False):
        (_, type_names, path, filename) = event

        print("PATH=[{}] FILENAME=[{}] EVENT_TYPES={}".format(
              path, filename, type_names))

if __name__ == '__main__':
    _main()

Steps to reproduce:

mkdir a  
rmdir a  
mkdir a  
mv a b

`superficial` parameter not forwarded to remove_watch_with_id()

remove_watch() accepts superficial=False, but does nothing with it. It looks like the author intended to forward it onto self.remove_watch_with_id(wd). Not really a big deal, since defaulting to False causes it to be removed from the OS event queue. But, it should probably either be passed, or removed from the signature.

Ignores IN_Q_OVERFLOW events

We are using this PyInotify on large directory trees holding millions of files with the possibility of large batches of file additions or deletions at any time. By keeping the kernel-level queue size large enough we generally avoid IN_Q_OVERFLOW events from the kernel. However, we've been surprised a few times and in those cases we lose file events with no warning.

It would be good to add explicit handling of the IN_Q_OVERFLOW event. In 0.2.7 if the kernel generates IN_Q_OVERFLOW, I believe that __handle_inotify_event immediately returns, event_gen's loop terminates, and then yields a final None event without explanation.

I haven't tried 0.2.8 yet, but it appears that the IN_Q_OVERFLOW event is simply ignored and operation continues. Assuming this is the case, this is a step backward from the standpoint of IN_Q_OVERFLOW.

I can work on this, but would like to get advice on what to do when it occurs. Options:

  • yield an event with no path, wd=-1, and the IN_Q_OVERFLOW mask
  • raise an exception

What do you think?

Thanks, Kim

Does this project have any active maintainers?

Browsing through the Issues and Pull Requests, I can see quite a few easy-to-fix problems, including PRs that address known issues with well-written patches. However, it seems the last commit was mid-2018.

@dsoprea - how do you feel about adding another maintainer? It would be great to flush a few of these waiting PRs.

Just for reference, this module is:

  • The first result on Google for "python inotify"
  • The first result on PyPI for "inotify"
  • Used by several large open-source projects - Odoo, Yelp's Paasta, Raspiblitz, to name a few.
  • Probably used in many other private projects.

AttributeError: module 'select' has no attribute 'epoll'

  File "/usr/local/lib/python3.6/site-packages/inotify/adapters.py", line 44, in __init__
    self.__epoll = select.epoll()
AttributeError: module 'select' has no attribute 'epoll'

and select from:

<module 'select' from '/tmp/tmp36/lib/python3.6/lib-dynload/select.cpython-36m-darwin.so'>

InotifyError should have more information as to what happened

[[sorry, re-raising the issue under my correct user]]

For example, add_watch raises InotifyError if the given file does not exist. It would be useful to get some more information as to what happened (file path, type of error - does not exists, cannot read, etc).

example (in this case the file does not exist):
DEBUG:inotify.adapters:Adding watch: [/tmp/leases.db] Error: unable to open file: ('Call failed (should not be -1): (-1) ERRNO=(0)',)

KeyError in __handle_inotify_event

Hi, I've recently started to use your nice little project in version 0.2.3 (I used pyinotify before).
However, I stumbled across the following error and hope that you can help me.

Traceback (most recent call last):
  File \"test.py\", line 88, in <module>
    sys.exit(main())
  File \"test.py\", line 82, in main
    handle_events(path)
  File \"test.py\", line 49, in handle_events
    for event in notifier.event_gen():
  File \"/foo/bar/lib/python2.7/site-packages/inotify/adapters.py\", line 193, in event_gen
    for event in self.__i.event_gen():
  File \"/foo/bar/lib/python2.7/site-packages/inotify/adapters.py\", line 148, in event_gen
    in self.__handle_inotify_event(fd, event_type):
  File \"/foo/bar/lib/python2.7/site-packages/inotify/adapters.py\", line 134, in __handle_inotify_event
    path = self.__watches_r[header.wd]
KeyError: 226

The number varies. I've seen all from 9 to 10 and from 220 to 229.
Have you seen this error before and can advice?

The function handle_events(path) is intended to announce all files under a path whenever they are closed after writing. It is defined as follows.

def handle_events(path):

    # we intend to react on files being closed after writing and directories being created
    mask = inotify.constants.IN_CLOSE_WRITE | inotify.constants.IN_CREATE

    # block_duration_s is the pause [s] between calls to the kernel
    # cycle: (poll for events, yield events sequentially, yield None, sleep for block_duration_s, loop)
    notifier = inotify.adapters.InotifyTree(path=path, mask=mask, block_duration_s=1)

    for event in notifier.event_gen():
        if event is None:
            continue

        (header, _type_names, watch_path, filename) = event

        # InotifyTree gives IN_ISDIR, IN_CREATE, IN_DELETE regardless of the actual mask for technical reasons
        # we ignore everything not in our mask
        if not header.mask & mask:
            continue

        # a file has been closed after writing. something to announce.
        if header.mask & inotify.constants.IN_CLOSE_WRITE:
            fpath = os.path.join(watch_path, filename)
            announce_file(fpath)

        # inotify adds a watch for the newly created path automatically (and removes it if the path is deleted)
        # but there might already be some files in the folder before the watch is active. let's make sure to announce them
        if header.mask & inotify.constants.IN_CREATE & inotify.constants.IN_ISDIR:
            announce_all(watch_path)

Best regards,
Tim

InotifyError should have more information about what happened

For example, add_watch raises InotifyError if the given file does not exist. It would be useful to get some more information as to what happened (file path, type of error - does not exists, cannot read, etc).

example (in this case the file does not exist):
DEBUG:inotify.adapters:Adding watch: [/tmp/leases.db] Error: unable to open file: ('Call failed (should not be -1): (-1) ERRNO=(0)',)

Inotify.__handle_inotify_event raises another TypeError when attempting to handle an event in python3.

Having fixed the two problems recently reported, another TypeError is raised while handling an event.

Traceback (most recent call last):
File "/inotify/dev/test.py", line 50, in
_main()
File "/inotify/dev/test.py", line 38, in _main
for event in i.event_gen():
File "/inotify/inotify/adapters.py", line 162, in event_gen
in self.__handle_inotify_event(fd, event_type):
File "/inotify/inotify/adapters.py", line 141, in __handle_inotify_event
filename = filename.rstrip('\0')
TypeError: 'str' does not support the buffer interface

Here, the problem is that filename is a 'bytes' object in python 3, so needs to be decoded before stripping. My fix was this:

filename = filename.encode(sys.getdefaultencoding()).rstrip('\0')

Add support/handling for inotify flags

We might want to add support for these flags as higher-level arguments to add_watch:

  • IN_DONT_FOLLOW
  • IN_EXCL_UNLINK

We should add support for doing updates to an existing watch (using IN_MASK_ADD). This prevents there being gaps in coverage when deleting and re-adding.

Exception if file path contains UTF-8 (EX:Chinese word)

Create folder with Chinese name. Then exception during file path encode then inotify watch stop.

[Exception Message]
adapters add_watch Adding watch: [/share/Music/新增資料夾]
Exception : argument 2: <type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode characters in position 13-17: ordinal not in range(128)
adapters del Cleaning-up inotify.

how is filter_predicate intended to be used?

Inotify class function event_gen excerpt:

                    for type_name in type_names:
                        if filter_predicate is not None and \
                           filter_predicate(type_name, e) is False:
self.__last_success_return = (type_name, e)

its expecting it to be callable?

Context Manager?

I find it a bit wonky that closing the watch can only be done through the destructor. Maybe it isn't the end of the world, but this sorta locks it to cPython.

Events not firing for os created directories

I modified the test script to monitor network connection file creation and deletion and the events aren't firing. I tested with a different user directory and it worked. This leads me to believe it is a permissions issue. Any thoughts on how to make this work?

`#!/usr/bin/env python3
import time
import os
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class Watcher:
DIRECTORY_TO_WATCH = '/proc/sys/net/ipv4/conf/'

def __init__(self):
    self.observer = Observer()

def run(self):
    event_handler = Handler()
    self.observer.schedule(event_handler, self.DIRECTORY_TO_WATCH, recursive=True)
    self.observer.start()
    try:
        while True:
            time.sleep(1)
            print('Checking...')
    except Exception as e:
        self.observer.stop()
        print("Error" + str(e))

    self.observer.join()

class Handler(FileSystemEventHandler):

@staticmethod
def on_any_event(event):
    print('Event triggered')
    if event.event_type == 'created':
        print(path + ' created')
    elif event.event_type == 'deleted':
        print(path + ' deleted')

if name == 'main':
w = Watcher()
w.run()`

inotify.adapters.InotifyTree doesn't update watch on rename folder

If you watch a folder and you rename the folder within that folder, event will not update the watch, if will keep on using old watch_path and filenames changed in that watch path will use old watch path:

Here is Renaming of the folder:
DEBUG 2016-08-24 16:35:13,717 (_INOTIFY_EVENT(wd=1, mask=1073741888, cookie=174440, len=16), ['IN_MOVED_FROM', 'IN_ISDIR'], b'/shared/media/content/Movies', b'VBH Demo')
DEBUG 2016-08-24 16:35:13,717 (_INOTIFY_EVENT(wd=1, mask=1073741952, cookie=174440, len=32), ['IN_MOVED_TO', 'IN_ISDIR'], b'/shared/media/content/Movies', b'VBH Demo Renamed')
DEBUG 2016-08-24 16:35:13,717 (_INOTIFY_EVENT(wd=2, mask=2048, cookie=0, len=0), ['IN_MOVE_SELF'], b'/shared/media/content/Movies/VBH Demo', b'')

And here is toucing the file in that new folder:
DEBUG 2016-08-24 16:36:55,914 (_INOTIFY_EVENT(wd=2, mask=256, cookie=0, len=16), ['IN_CREATE'], b'/shared/media/content/Movies/VBH Demo', b'TESTFILE')
DEBUG 2016-08-24 16:36:55,915 (_INOTIFY_EVENT(wd=2, mask=32, cookie=0, len=16), ['IN_OPEN'], b'/shared/media/content/Movies/VBH Demo', b'TESTFILE')
DEBUG 2016-08-24 16:36:55,915 (_INOTIFY_EVENT(wd=2, mask=4, cookie=0, len=16), ['IN_ATTRIB'], b'/shared/media/content/Movies/VBH Demo', b'TESTFILE')
DEBUG 2016-08-24 16:36:55,915 (_INOTIFY_EVENT(wd=2, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], b'/shared/media/content/Movies/VBH Demo', b'TESTFILE')

As you can see, watch path is not changed to VBH Demo Renamed, it stayed in VBH Demo.

Documentation has unnecessary/misleading warning

At the end of the readme, it says:

Python's VM will remain locked and all other threads in your application will cease to function until something raises an event in the directories that are being watched.

This is actually completely untrue. You are using Python's select.epoll interface to wait for events, which already handles releasing the GIL internally, so it will not block other threads while it is waiting. I have tested and confirmed that using this inotify library with other threads running works just fine, and does not block other threads or "lock the VM" in any way..

(Given this, the whole recommendation about using multiprocessing instead of threads is really unnecessary. If somebody doesn't already have other reasons to use multiprocessing, it would just add needless extra overhead in this case.)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.