FileNotFoundError about delta-rs HOT 6 CLOSED

looks like a duplicate to #72, should be fixed by #73.

from delta-rs.

@martindut fix as been merged into master, please reopen if you are still experiencing this issue with 0.2.1 release in PyPI.

from delta-rs.

@houqp , now I'm getting this error if I run df = dt.to_pyarrow_table():
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.8/site-packages/deltalake/__init__.py", line 38, in to_pyarrow_table return self.to_pyarrow_dataset().to_table() File "/usr/local/lib/python3.8/site-packages/deltalake/__init__.py", line 33, in to_pyarrow_dataset return dataset(keys, filesystem=f"{paths[0].scheme}://{paths[0].netloc}") File "/usr/local/lib/python3.8/site-packages/pyarrow/dataset.py", line 674, in dataset return _filesystem_dataset(source, **kwargs) File "/usr/local/lib/python3.8/site-packages/pyarrow/dataset.py", line 426, in _filesystem_dataset fs, paths_or_selector = _ensure_multiple_sources(source, filesystem) File "/usr/local/lib/python3.8/site-packages/pyarrow/dataset.py", line 298, in _ensure_multiple_sources filesystem, is_local = _ensure_fs(filesystem) File "/usr/local/lib/python3.8/site-packages/pyarrow/dataset.py", line 232, in _ensure_fs filesystem, prefix = FileSystem.from_uri(fs_or_uri) File "pyarrow/_fs.pyx", line 347, in pyarrow._fs.FileSystem.from_uri File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status pyarrow.lib.ArrowInvalid: Unrecognized filesystem type in URI: abfss://[email protected]

from delta-rs.

@martindut I think our integration tests aren't covering the use of pyarrow and the Azure storage engine. Would you mind opening a new issue with some details?

from delta-rs.

@houqp I think this is still an issue. I am on version

deltalake-0.4.8 numpy-1.20.3 pyarrow-4.0.0

When I run:

from deltalake import DeltaTable
import os

os.environ['AZURE_STORAGE_ACCOUNT']='xxx'
os.environ['AZURE_STORAGE_KEY']='xx'

dt = DeltaTable('abfss://[email protected]/delta_example/')
print(f'table version: {dt.version()}')
print(f'list of files: {dt.file_paths()}')

# convert to data.frame
df = dt.to_pyarrow_table().to_pandas()

The dt.to_pyarrow_table() causes the following error:

ArrowInvalid: Unrecognized filesystem type in URI: abfss://[email protected]

from delta-rs.

@samuel100 are you able to read one of the parquet file that's causing the error with pyarrow directly? I don't have an azure environment to test and debug this at the moment.

from delta-rs.