alexdelorenzo / aiopath Goto Github PK

Would you be open to re-license to a more permissive license, like MIT or BSD?

pathlib.Path.samefile() accepts Path objects and strs, AsyncPath.samefile() doesn't.

Would you consider using a different software license? This library is fantastic. However, it is not possible to use at some companies which prohibit the use of the LGPL license. I completely respect the decision to use LGPL if it accurately reflects the intentions of this project.

However, if another license such as MIT or BSD would be acceptable, then it would be possible for more companies to use this package as part of private projects.

I am not a lawyer and I am not one of those people who are passionately versed in the details of open source licensing and philosophy. I am just an engineer who likes using your package and needs to comply with corporate software policies. Therefore, I probably won't be able to help much with answering any detailed questions about why one license is better than another. I respect the decision of the maintainers regardless of which way you go.

Hi,
I've find a possible bug in AsyncPath lib:
f.read(count) doesn't advacne offset, so two consecutive reads read just the beggining of the file.
I've created tests showing the bug:

from pathlib import Path

import pytest
from tempfile import TemporaryDirectory
from aiopath import AsyncPath


@pytest.mark.asyncio
async def test_async_path():
    with TemporaryDirectory(prefix='read_test') as temp_dir:
        file = AsyncPath(temp_dir) / 'test.bin'
        await file.write_bytes(b'0123456789abcdefghijklmnopqrstuvwxyz')
        async with file.open('rb') as f:
            chunk = await f.read(10)
            assert chunk == b'0123456789'
            chunk = await f.read(10)
            assert chunk == b'abcdefghij' # THIS LINE FAILS - eventhough it shouldn't (in pathlib this works)


def test_path():
    with TemporaryDirectory(prefix='read_test') as temp_dir:
        file = Path(temp_dir) / 'test.bin'
        file.write_bytes(b'0123456789abcdefghijklmnopqrstuvwxyz')
        with file.open('rb') as f:
            chunk = f.read(10)
            assert chunk == b'0123456789'
            chunk = f.read(10)
            assert chunk == b'abcdefghij'

typing on PyPI hasn't been updated since Python 3.7 was released. aiopath uses typing features from the standard library that only exist in Python 3.8+.

typing_extensions is the PyPI package to use.

Use of the wrong package is the cause of issue #3.

I found that there was another similar project to this called aiopathlib which is now deprecated in favor of anyio.Path. Are there any particular pros and cons of this project vs anyio.Path?

The AsyncPath.walk method does not return an AsyncGenerator as expected, but instead a synchronous Generator object. This appears to be because of .walk not being defined in AsyncPath and thus the method from Path is used instead.

Expected Behaviour

AsyncPath.walk should return an AsyncGenerator object that can be used in async for statements and implements __aiter__ and __anext__.

Actual Behaviour

AsyncPath.walk follows the MRO and is really Path.walk, thus returning a normal generator. Using it in async for will cause a TypeError.

>>> async for p in AsyncPath(".").walk():
...     print(p)
... 
Traceback (most recent call last):
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "<console>", line 1, in <module>
TypeError: 'async for' requires an object with __aiter__ method, got generator

Steps to reproduce

1. Install aiopath
2. In a console, IDLE, or some program of your choice that runs python enter the following:

from aiopath import AsyncPath
print(*AsyncPath(".").walk())

3. See that there was an output instead of the error that should have occured.

I'm trying to package aiopath for NixOS. This requires the source distribution to include requirements.txt (similar to nix-community/pypi2nix#456 ).

aiopath's Pypi distribution does not include requirements.txt

$ curl -s 'https://files.pythonhosted.org/packages/4e/ab/04f87a414f227d92bc1278202356aea71ebdb1d75f31f16b1ad057c70c44/aiopath-0.5.11.tar.gz' | tar ztf -
aiopath-0.5.11/
aiopath-0.5.11/LICENSE
aiopath-0.5.11/PKG-INFO
aiopath-0.5.11/README.md
aiopath-0.5.11/aiopath/
aiopath-0.5.11/aiopath/__init__.py
aiopath-0.5.11/aiopath/flavours.py
aiopath-0.5.11/aiopath/handle.py
aiopath-0.5.11/aiopath/path.py
aiopath-0.5.11/aiopath/scandir.py
aiopath-0.5.11/aiopath/selectors.py
aiopath-0.5.11/aiopath/types.py
aiopath-0.5.11/aiopath/wrap.py
aiopath-0.5.11/aiopath.egg-info/
aiopath-0.5.11/aiopath.egg-info/PKG-INFO
aiopath-0.5.11/aiopath.egg-info/SOURCES.txt
aiopath-0.5.11/aiopath.egg-info/dependency_links.txt
aiopath-0.5.11/aiopath.egg-info/requires.txt
aiopath-0.5.11/aiopath.egg-info/top_level.txt
aiopath-0.5.11/aiopath.egg-info/zip-safe
aiopath-0.5.11/pyproject.toml
aiopath-0.5.11/setup.cfg
aiopath-0.5.11/setup.py

You mpris_server does, for example:

$ curl -s 'https://files.pythonhosted.org/packages/9c/0c/bd61ff9584f4597828bee8538a68be08cd5ac60c8c0f80414c7f4b36862c/mpris_server-0.4.2.tar.gz' | tar ztf -
mpris_server-0.4.2/
mpris_server-0.4.2/LICENSE
mpris_server-0.4.2/MANIFEST.in
mpris_server-0.4.2/PKG-INFO
mpris_server-0.4.2/README.md
mpris_server-0.4.2/mpris_server/
mpris_server-0.4.2/mpris_server/__init__.py
mpris_server-0.4.2/mpris_server/adapters.py
mpris_server-0.4.2/mpris_server/base.py
mpris_server-0.4.2/mpris_server/events.py
mpris_server-0.4.2/mpris_server/interfaces/
mpris_server-0.4.2/mpris_server/interfaces/__init__.py
mpris_server-0.4.2/mpris_server/interfaces/interface.py
mpris_server-0.4.2/mpris_server/interfaces/player.py
mpris_server-0.4.2/mpris_server/interfaces/playlists.py
mpris_server-0.4.2/mpris_server/interfaces/root.py
mpris_server-0.4.2/mpris_server/interfaces/tracklist.py
mpris_server-0.4.2/mpris_server/mpris/
mpris_server-0.4.2/mpris_server/mpris/__init__.py
mpris_server-0.4.2/mpris_server/mpris/compat.py
mpris_server-0.4.2/mpris_server/mpris/metadata.py
mpris_server-0.4.2/mpris_server/server.py
mpris_server-0.4.2/mpris_server/types.py
mpris_server-0.4.2/mpris_server.egg-info/
mpris_server-0.4.2/mpris_server.egg-info/PKG-INFO
mpris_server-0.4.2/mpris_server.egg-info/SOURCES.txt
mpris_server-0.4.2/mpris_server.egg-info/dependency_links.txt
mpris_server-0.4.2/mpris_server.egg-info/requires.txt
mpris_server-0.4.2/mpris_server.egg-info/top_level.txt
mpris_server-0.4.2/mpris_server.egg-info/zip-safe
mpris_server-0.4.2/pyproject.toml
mpris_server-0.4.2/requirements.txt
mpris_server-0.4.2/setup.cfg
mpris_server-0.4.2/setup.py

Any plan to support python >=3.7 ?

setup.py reads requirements.txt to populate install_requires. The issue is that typically requirements.txt requirements are locked to specific versions for reproducibility of developer environments. This is not ideal for install_requires since these specific version dependencies will reduce the entire environment of someone who installs this package.

For example, once anyio 3.2 comes out, I have to wait for this project to update requirement.txt before I can use it in my project.

The solution is to update setup.py to be less specific about version numbers:

install_requires=["anyio>=3,<4", "aiofile>=3,<4"]

It would be great to create a conda-forge package.

In 3.7.0 the _ignore_error function doesn't exist in pathlib, so aiopath fails to import.

Adding a try/except block around this import that on except executes the following from cpython would solve this issue.

_WINERROR_NOT_READY = 21  # drive exists but is not accessible
_WINERROR_INVALID_NAME = 123  # fix for bpo-35306
_WINERROR_CANT_RESOLVE_FILENAME = 1921  # broken symlink pointing to itself

_IGNORED_ERRNOS = (ENOENT, ENOTDIR, EBADF, ELOOP)

_IGNORED_WINERRORS = (
    _WINERROR_NOT_READY,
    _WINERROR_INVALID_NAME,
    _WINERROR_CANT_RESOLVE_FILENAME)

def _ignore_error(exception):
    return (getattr(exception, 'errno', None) in _IGNORED_ERRNOS or
            getattr(exception, 'winerror', None) in _IGNORED_WINERRORS)

This would have to be in a 0.5.13 release to support older pythons. I'll happily submit a PR for this.

Instead of using asyncio, move to anyio for asyncio, trio and curio support.

https://anyio.readthedocs.io/
https://github.com/python-trio/sniffio

Receives bytes, but can also give to memoryview. (buffer.getbuffer() -> memoryview)

But pycharm will show a warning.

source code https://github.com/alexdelorenzo/aiopath/blob/main/aiopath/path.py#L180

# type-check for the buffer interface before truncating the file
view = memoryview(data)

I don't quite understand what this means.

I originally used getvalue, but after reading this https://stackoverflow.com/questions/61319551/when-should-one-use-bytesio-getvalue-instead-of-getbuffer, it seems that getbuffer would be better.

Everything works, it's just that I don't want pycharm to have a warning, I just want to know, thanks.

async def resize(path, size: Tuple[int, int] = (200, 200)):
    img = Image.open(path)
    buffer = io.BytesIO()
    if img.mode == 'P':
        img = img.convert('RGB')
    img.thumbnail(size=size)
    img.save(buffer, format=img.format.lower())
    await AsyncPath(path).write_bytes(buffer.getbuffer())

Python 3.11 appears to have removed a private class this library depends on:

Traceback (most recent call last):
  File ".../myapp.py", line 6, in <module>
    from aiopath import AsyncPath
  File ".../.venv/lib/python3.11/site-packages/aiopath/__init__.py", line 2, in <module>
    from .path import AsyncPath, AsyncPurePath, AsyncWindowsPath, \
  File ".../.venv/lib/python3.11/site-packages/aiopath/path.py", line 2, in <module>
    from pathlib import PosixPath, WindowsPath, _NormalAccessor, \
ImportError: cannot import name '_NormalAccessor' from 'pathlib' (~/.pyenv/versions/3.11.0rc2/lib/python3.11/pathlib.py)

path.py line 675: self._accessor spelled with only one "c".

on python 3.12 (at least, have not tested 3.11), using rglob() leads to this:

/usr/lib/python3.12/pathlib.py:169: RuntimeWarning: coroutine 'AsyncPath.is_dir' was never awaited
  if not parent_path.is_dir():
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

sample code:

async def _list_directory_async_recursive(path: str, mask: str = None, files_only: bool = False) -> list[str]:
    paths = []
    if mask is None:
        mask = '*'

    p: AsyncPath = AsyncPath(path)
    async for pp in p.rglob(mask):
        if files_only and await pp.is_dir():
            # skip directories
            continue
        paths.append(str(pp))

    return paths

glob() seems unaffected.

a simple fix is, in aiopath.path.py, to change this:

async def rglob(self: Self, pattern: str, *, case_sensitive: bool | None = None) -> AsyncIterable[Self]:
    for path in await to_thread(super().rglob, pattern, case_sensitive=case_sensitive):
      yield AsyncPath(path)

to this

async def rglob(self: Self, pattern: str, *, case_sensitive: bool | None = None) -> AsyncIterable[Self]:
    for path in await to_thread(self._path.rglob, pattern, case_sensitive=case_sensitive):
      yield AsyncPath(path)

if you agree, i could make a PR. i also see the same pattern with super() is used in other functions in aiopath.path.py, though i have not tested if they have the same issue.

Tested with python 3.8 and Python3.9. Works ok on aiopath 0.5.6

async for path in AsyncPath('.').glob():
    print(path)
TypeError: func_to_async_func() takes 1 positional argument but 2 were given

await f.read_text()
    async with self.open('r', encoding=encoding, errors=errors) as file:
AttributeError: __aexit__

The following works on alpha 6 and below, but alpha 7 introduces this error:

In [1]: from aiopath import AsyncPath                                                                                                                                  
                                                                                                                                                                       
In [2]: home = await AsyncPath.home()                                                                                                                                  
---------------------------------------------------------------------------                                                                                            
TypeError                                 Traceback (most recent call last)                                                                                            
<ipython-input-2-ef16332cd56b> in <module>                                                                                                                             
----> 1 home = await AsyncPath.home()                                                                                                                                  
                                                                                                                                                                       
~/.pyenv/versions/3.10-dev/lib/python3.10/site-packages/aiopath/path.py in home(cls)                                                                                   
    391     returned by os.path.expanduser('~')).                                                                                                                      
    392     """                                                                                                                                                        
--> 393     coro = cls()._flavour.gethomedir(None)                                                                                                                     
    394     homedir: str = await coro                                                                                                                                  
    395                                                                                                                                                                
                                                                                                                                                                       
~/.pyenv/versions/3.10-dev/lib/python3.10/site-packages/aiopath/path.py in __new__(cls, *args, **kwargs)                                                               
    133       cls = AsyncWindowsPath if os.name == 'nt' else AsyncPosixPath                                                                                            
    134 
--> 135     self = cls._from_parts(args, init=False)
    136 
    137     if not self._flavour.is_supported:

TypeError: PurePath._from_parts() got an unexpected keyword argument 'init'

When reading an NTFS file system on windows, a program may encounter junction points, which are similar to symbolic links, but a bit different, see this GeeksforGeeks page https://www.geeksforgeeks.org/ntfs-junction-points/. The readlink functions from pathlib.Path and aiopath.AsyncPath act slightly differently when encountering junction points. When calling Path.readlink() on a junction point, the path the junction point points to is returned. AsyncPath.readlink() instead raises a TypeError.

For testing purposes, you can use the junction point C:\Users\[YOUR USERNAME]\AppData\Local\Application Data\ which points to C:\Users\[YOUR USERNAME]\AppData\Local\.

Hi @alexdelorenzo,

thanks for this handy library. Delving into the code I cannot see the necessity for anyio dependency.

stdlib asyncio provides to_thread or loop.run_in_executor to push sync functions to threads.
What is the motivation to using AnyIO?

Maybe aiofiles can be demoted to dev dependency, as it is only used by tests. To clean the installation procedure we could remove setup.py and requirements*.txt to use only pyproject.toml / poetry.

Would a PR be welcome?

Thanks and merry christmas!

M

I was thrilled to see that you're using type annotations, but mypy doesn't like it because you're missing py.typed in the package.
Can you add it so that mypy will scan for your annotations?

Using aiopath 0.6.11 in a Starlette project. Used await path.resolve() Think this is on you:

<PROJECT>/.venv/lib/python3.11/site-packages/aiopath/path.py:571 in resolve
   568 │
   569 │   # Now we have no symlinks in the path, it's safe to normalize it.
   570 │   normed: str = self._flavour.pathmod.normpath(s)
 ❱ 571 │   obj = self._from_parts((normed,), init=False)
   572 │   obj._init(template=self)
   573 │   return obj
TypeError: PurePath._from_parts() got an unexpected keyword argument 'init'

First of all, thanks for this lib!
You made an awesome job porting the pathlib interface to async =)

I have a question about your readme, under the "Implementation" section there is a paragraph that mentions the following.

aiopath is typed with Python type annotations, and if using the aiofile back end, it takes advantage of libaio for async I/O on Linux.

I didn't get the part "if using the aiofile", is there any other option besides aiofile?

As a fix for #13, IterableAIOFile overrides the read()/write() methods in order to keep track of the file cursor's offset. While there's a lock in the parent classes read()/write() methods, no lock is held when managing the offset.

Two coroutines with access to the same IterableAIOFile can end up with a data race if they call either of those methods concurrently.

Simple fix is to enter a shared asyncio.Lock object before reading, writing or managing the file cursor offset.